Category Archives: Disaster

Post navigation

I was going through my old drafts and found this post I had started to write earlier this year but never finished. Actually it appears I meant this to be part of White (K)nights but I cut it out to make that post more readable.

During my media interactions I was asked multiple times to comment on Elon Musk and once or twice on his submarine. I tried to keep my comments fairly neutral, but the truth is, I and some of my fellow trained cave rescuers were pretty bothered by Musk’s attempted involvement. I got into at least one online debate about how the people in charge obviously were clueless and that Musk’s solution of a submarine was a brilliant idea.

It wasn’t and I figured I’d address some of my concerns. Please note as with all situations like this, I was not directly involved, so I’m going on publicly available facts and my training as a cave rescue person and a cave rescue instructor. I am also not in any way speaking on behalf of the National Cave Rescue Commission or the NSS.

Now let’s discuss the device itself:

It almost certainly would not have fit. By all accounts, the tightest pinch was 15″ and hard to navigate. Anyone who has moved through a cave knows that even larger passages can be hard to navigate. Locally we have a cave that has a pinch that’s probably close to 15″, but that is at the bottom of a body sized V-shaped passage. Unless you can bend in the middle, you will not fit through it. A cylinder like Musk designed, would not fit. I don’t know the passages in the Thai cave, but odds are there is more than one passage where flexibility is important.

It also, in many ways was superbly dangerous. Once sealed into the tube, there would be no easy way to monitor the patient’s vitals. And if the tube had started to leak (cave environments can be extremely destructive, even to metal objects), there appears there would have been no recourse except to keep swimming and hoping to get to an air filled chamber quickly enough and that was large enough to debug the issue.

In addition, if the patients were not sedated, I’d have to imagine that being sealed into such a tube, even with lights for 20-40 minutes at a time would have been sheer terror. As it is, the kids were in fact apparently heavily sedated (a fact that some of us still find a bit surprising, even though very understandable), and yet at least one started to come out of sedation while in a water passage. Without being able to directly monitor the vitals of the patient, who knows what would have happened.

There’s probably other issues I could come up with. But let me end with this one. Rarely if ever do you want to beta-test or heck even alpha-test, which is what this would have been, a brand new design in a life or death situation when there are alternatives.

Like our White Knights, we want our brilliant tech solutions, but often we’re better off adapting what we’ve done in the past. In cave rescue we try to teach our students a “bag of tricks” that they can adapt to each particular rescue. Foe example, there is no single rigging solution that will work for every rescue. How I might rig a drop in Fantastic in Ellison’s might be very different from how I’d rig a drop here in New York. How I package a patient for movement here may be different than in a Puerto Rican cave. And honestly I’ve seen a lot of high-tech equipment get suggested for cave rescue that simply doesn’t work well in a cave environment and we often go back to the simple proven stuff.

I will add a tease, to perhaps a future blog post, of a mock rescue rescue where a high-tech approach failed after several hours of trying, and the low-tech solution solved the problem.

When Columbia was lost on re-entry a lot of folks came out of the woodwork to proclaim that Soyuz was obviously so much safer since no crew had died the ill-fated Soyuz 11 flight in 1971. The problem with this line of thought was that at the time of Columbia, Soyuz had only flown 77 times successfully vs 89 successful flights since the Challenger Disaster. So which one was safer? If you’re going strictly on the successful number of flights, the Space Shuttle. Of course the question isn’t as simple as that. Note I haven’t even mentioned Soyuz 1, which happened before Soyuz 11 and was also a fatal flight.

Some people tried to argue that the space shuttle was far less safe because during the program it had killed 14 people during its program life vs 4 for Soyuz. I always thought this was a weird metric since it all came down to the number of people on board. Had Columbia and Challenger only flown with 2 on each mission, would the same folks argue they were equally safe as Soyuz?

But we can’t stop there. If we want to be fair, we have to include Soyuz-18a. This flight was aborted at a high altitude (so technically they passed the Karman Line and are credited with attaining space.) Then in 1983, Soyuz T-10a also suffered an abort, this time on the pad.

So at this point I’m going to draw a somewhat arbitrary line as to what I consider a successful mission: the crew obtains an orbit sufficient to carry out a majority of their planned mission and returns safely. All the incidents above, Soyuz and Space Shuttle are failed missions. For example, while Soyuz-11 and Columbia attained orbit and carried out their primary missions, they failed on the key requirement to return their crew safe.

Using that definition, the shuttle was far more successful. There was one shuttle flight that did undershoot the runway at Edwards, but given the size of the lakebed, landed successfully. We’ll come back to that in a few.

Now let me add a few more issues with the Soyuz.

Soyuz-5 – failure of service module to separate, capsule entered upside-down, and the hatch nearly burned through. The parachute lines also tangled resulting in a very hard landing.

TMA-10 – Failure of the Service Module to separate caused the capsule to re-enter in an improper orientation (which could have lead to a loss of the crew and vehicle) which ended up causing the capsule also re-enter in a ballistic re-entry mode. The Russians initially did not tell the US.

TMA-11 – Similar issue as TMA-10, with damage to the hatch and antenna that was abnormal.

And there have been others of varying degree. I’m also ignoring the slew of Progress failures, including the 3 more recent ones that were launched on a rocket very similar to the current Soyuz-FG.

So, what’s safer, the Soyuz or the Space Shuttle? Honestly, I think it’s a bit of a trick question. As one of my old comrades on the Usenet Sci.space.* hierarchy once said, “any time a single failure can make a significant change in the statistics, means you really don’t have enough data.” (I’m paraphrasing).

My personal bias is, both programs had programmatic issues (and I think the Russians are getting a bit sloppier when it comes to safety) and design issues (even a perfectly run shuttle program had risks that could not have been solved, even if they might have prevented both Challenger and Columbia). However, I think the Russian Soyuz is ultimately more robust. It appears a bit more prone to failures, but it has survived most of them. But, that still doesn’t make it 100% safe. Nor does it need to be 100% safe. To open the new frontier we need to take some risks. It’s a matter of degree.

“A ship in harbor is safe, but that is not what ships are built for.” – John A. Shedd.

A spacecraft is safe on the ground, but that’s not what it’s built for.

In the meantime, there’s a lot of, in my opinion naive, talk about decrewing ISS. I suspect the Russians will fly the Soyuz TM-11 flight as scheduled. There’s a slight chance it might fly uncrewed and simply serve to replace the current Soyuz TM-9 capsule, but it will fly.

We all know the story of the boy who cried wolf. Last week we had a nationwide example of that.

I’m about to break an unwritten rule I have for this blog in that I try to avoid politics as much as possible. But here I’m going to try to steer away from any particular partisan position and try to discuss the impact of both certain policies and the resulting reactions.

So, to be upfront, I am not a fan of President Trump, nor do I subscribe to his brand or style of politics. That said, let’s carry on.

So, at approximately 2:20 PM EDT on Thursday of last week, millions of Americans had their phones buzz, beep, play some sort of tune, etc. By the build up and reaction, you would have thought it was the end of the world. Ironically, the system MIGHT someday be used to actually alert us to the end of the world. Hopefully not.

The event I’m referring to of course was a test of a new system that many phones classify as a “Presidential Alert”. It’s really the latest in a series of systems the US has had over the years to alert citizens to potential dangers or crises.

Some of my readers may be old enough to recall AM radios that had two markings on them, small triangles with a CD in them. This was for the CONELRAD alert system that was in place from 1953-1963. This was designed to be used strictly in the event of a nuclear attack and was never intended nor used in the event of a natural disaster.

There are, and have been from day one of the design for phones, three types of alerts, the “Presidential Alert”, alerts for extreme weather or other events and Amber alerts. Phones have had the ability to receive these alerts for close to a decade now; and, importantly, for the second two type of alerts, the ability to shut them off. Phones can NOT turn off the Presidential Alert. This is by design and this has been a feature of the system from day 1. In other words, despite what many in social media seemed to believe, this feature was baked in long before President Trump took office.

So enough history, let’s get to the the wolf cry. Both before and after, I saw people all over Facebook and other media proclaiming how bothered they were and upset that the President had the ability to text them directly. He (or ultimately she) can’t.

Ok, that’s not quite true. My understanding is that the President can issue a statement through the White House Communications Director that gets passed on to the appropriate people that would activate the EAS and the WEA and the statement would go out. But the idea of President Trump or any President sitting at their desk and picking up their phone and texting all of America is not true. It’s a myth and image built up by folks who are quite frankly paranoid. This does not mean that the system can’t be abused. However, there are numerous checks in the system that I’m extremely doubtful that such non-emergency use would ever actually intentionally occur.

But, the fact that people apparently feel so strongly about the risks troubles me. There’s no doubt that this President uses social media in ways unlike any previous President. This President is far more likely to say what’s on his mind without much filter. Some people love him for that, some vilify him.

BUT, this man is the President, NOT the Office of the President nor the entire Executive Branch. This is an important distinction and one to keep in mind. Regardless of how one feels about the State of the Union, there are still checks on the actual authority he can wield. And ultimately if the system did get abused, one would hope that someone along the chain would say “no” or if it got beyond that Congress would ultimately enact additional safeguards.

For a system like the EAS and the WEA to work, we need to test them. And we need to have faith they are properly used. Yes, sometimes mistakes happen in an unscheduled test going out, or worse, a test mistakenly sending out a message that a real event is transpiring. These mistakes NEED to be avoided and minimized so that people don’t panic (which can cause harm, including death in some cases). But the testing needs to happen to make sure the system DOES work when needed. We need to have a general faith, though perhaps tempered with SOME caution of abuse of the system. (BTW, I do realize there’s some controversy over exactly what transpired in the Hawaii incident and in fact might actually illustrate an actual abuse of the system by an individual.)

But we should not let the partisan social media actions of one particular President make us never believe the boy who cried wolf. Someday the cry may be real.

As long as the national level tests like the one that occurred last Thursday remain infrequent, with a clear purpose, and are clearly tests, I will continue to advocate for them.

P.S. Oh, one more addendum, anything you see about John McAfee concerning the test, or the E911 of your phone should be basically ignored.

P.P.S One of the eeriest experiences of my life was walking into my apartment and catching a rebroadcast of the movie Countdown to Looking Glass. It made me better understand how folks could have fallen for the Orson Welles broadcast of The War of the Worlds. Now I would never advocate searching for a bootleg copy of the movie on Youtube, but if you can find a copy it’s worth watching in my opinion, and honestly, the last minute or so still sort of freaks me out.

This is actually the name of an episode of Dirty Jobs. But it’s a title that has stuck with me because it’s near and dear to the sort of things I like to think about. Mike Rowe has a good follow-up article here. The title and show ruffled feathers, but he’s right, it’s an important concept to discuss.

You’ll often hear the mantra “Safety First”. This often means in work places things like wearing fall protection when working at height, or wearing a life vest when working in water, or ear protection, or other safety measures. The idea being that above all else, we have to be safe.

I got thinking about this while reading Rand Simberg’s book, Safe is Not an Option. He argues that trying to make safety the highest priority of spaceflight is holding us back. I tend to agree. And I’d like to argue out that despite NASA talking about safety in public announcements, the truth is NASA hasn’t always been upfront about it and also it has made decisions where safety wasn’t first (and I would argue in some cases those decisions were justified).

Now I know at least a few of my readers have read the Rogers Commission Report on the Challenger shuttle disaster. It’s worth the read, especially Dr. Feynman’s appendix. One of the issues that came up during the investigation was exactly how safe the Shuttle was. (Here I’m referring to the entire system, the orbiter, SRBs and ET). Some at NASA were claiming that the Shuttle had a 1 in 100,000 chance at a loss of an orbiter. (a loss of a an ET or SRB as long as it didn’t impact the Shuttle wasn’t really a concern, as all ETs were lost at the end of each mission and at least 2 SRBs were lost due to other issues). As Feynman pointed out, this meant you could fly the Shuttle every day for 300 years and only have one accident. What was the reasoning behind such an argument? Honestly, nothing more than wishful thinking. As we know, the shuttle was far less safe, 1 in 67.5. That’s a hugely different number.

There were many reasons that lead to either accident and I won’t delve into them here; though I would highly recommend The Challenger Launch Decision by Diane Vaughen as a comprehensive analysis of the decision making that helped lead up to the Challenger disaster.

But let’s talk a bit about how things could have been made safer, but NASA correctly decided NOT to go down that route. One early iteration of the shuttle design had additional SRBs mounted to the orbiter that would have been used to abort during an additional 30 seconds of the flight envelope1. I can’t determine if these 30 seconds would have overlapped with the critical 30 seconds Challenger’s final mission. But let’s assume they did. The total cost would have added $300 million to the development of the program and reduced the payload capacity of the orbiter2..

In a system already beset with cost considerations and payload considerations, this might have meant the program never got off the ground literally. Or if it did, it would fail to meet its payload guidelines. All this for 30 more seconds of additional safety. Would that have been worth it? Arguably not.

Another design decision was to eliminate thrust termination for the SRBs. Again, this is something that would have arguably made the ascent portion of the flight safer: in theory. The theory being that since you can’t normally shut down the SRBs, you can’t perform an orbiter separation, which means the orbiter can’t detach during the first 2 minutes of the flight and hence can’t perform a return to launch site abort.

But again, adding that safety feature didn’t necessarily make things better. For one thing, it really only would have been useful above a certain altitude since below that altitude all the orbiter could have done is detach from the stack and fallen into the sea with too little time to get into a glide position and make it back to a runway.

But there was a bigger issue: the thrust termination was determined to be violent enough it would probably have damaged the orbiter if used. This could have been mitigated by beefing up the orbiter structure. But this would have imposed an 8,000 lb payload penalty. Since the shuttle was already having trouble reaching its 65,000 lb payload goal, this was determined to be unacceptable3.

So, NASA could have made the decision of “safety first” and ended up with a shuttle system that never would have flown. And given the political calculus at the time, it’s unlikely NASA could have come up with a better solution nor had Congress fund it. The shuttle was an unfortunate compromise brought about a host of factors. But it did fly.

As I like to tie this back to some of my other interests; so what about caving and cave rescue.? I mentioned in a previous post how we’ve moved away from treating one line in the system strictly as a belay line. But what if I told you we often only use one line! There are many places in caving and cave rescue where we do not have a belay line. A good example is for a caver ascending or descending a rope. This is called Single Rope Technique or SRT. There are some who come to caving from other activities and ask “where’s your belay? You have to have a belay!”

But, a belay line (here used in the sense of catching a caver from a potentially dangerous fall if their mainline fails) is actually far less safe. I’ll give an example. First let’s start with some possible failure modes

Main rope being cut or damaged to the point of failure

The point the rope is rigged to (the anchor point) failing

Your ascent or descent system failing

So the idea is, if one of those 3 things happen, the belay line will catch you. But there’s issues with that theory. One major issue is that large drops in caves are often accompanied by air movement and waterfalls. The air movement, or even simple movements by the caver (and influenced by the rope in some cases) can cause a twisting motion. This means that before you know it, your belay line has been twisted around your mainline and you can no longer ascend or descend. You’re stuck. Now combine this with being in a waterfall and you’ve become a high-risk candidate for hypothermia, drowning, and harness hang syndrome. In other words, your belay line has now increased your chances of dying. So much for the attitude safety first.

Even if you avoid those issues, you haven’t really solved the possible failure modes I listed. If you think about it, anything that’s going to damage your mainline is possible to your belay line. There are some differences, your belay line, for example because it’s moving is far less likely to wear through in a single spot like a mainline might from being bounced on during an ascent. On the other hand it’s more possible to suffer a shock load over a sharp edge if it’s not attended well.

If your mainline anchor point fails, you’re relying on your belay anchor point to be stronger. If it’s stronger, why not use it for your mainline? (there are reasons not to, but this is a question that should cross your mind.)

Finally, for equipment failure, catastrophic failure is rare (only seen in movies honestly) and other failures are better mitigated by proper inspection of your equipment and close attention to proper technique.

Of course the safest thing to do, if we were really putting safety first would to never go caving. But where’s the fun in that.

We can insist on safety first in much of what we do, but if we do, we inhibit ourselves from actually accomplishing the activity and in some cases can actually make things LESS safe by trying to add more safety. And safety is more than simply adding additional pieces to a system. It’s often proper procedures. Rather than adding a belay line, focusing on better rigging and climbing technique for example. Or even simply accepting that sometimes things can go sideways and people may be injured or die. We live in a dangerous world and while we can make things safer and often should, we should be willing to balance our desire for safety with practicality and the desirability of the goal.

I’m going to end with two quotes from an engineer I respected greatly, Mary Shafer who formerly worked at NASA at what was Dryden Flight Research Center and is now the Armstrong Flight Research Center at Edwards Air Force Base.

Insisting on absolute safety is for people who don’t have the balls to live in the real world.

and

There’s no way to make life perfectly safe; you can’t get out of it alive.

For a more complete record of Mary’s thoughts, I direct you to this post.

Footnotes

There’s a military aphorism “Train as you fight, fight as you train.” I was recently reminded of this by a friend mine and a reader of my blog. We’ve shared a mutual interest in the space program for decades. He mentioned this last week (though I can’t seem to find the post) in response to something I wrote and it got me thinking.

When we teach cave rescue, we almost always use a real patient in the litter. There’s a couple of reasons for this. For one, it ipso facto recreates the actual mass and weight distribution of a real patient. Now, there are training dummies that are similar in weight and mass, but they can be a pain in the neck. For one thing, ever try to move an inert body? That’s what a training dummy can be like. Sure it’s great once it’s IN the litter, but getting it into position deep inside a cave can be almost impossible.

For another, it gives our students a chance to experience what being a patient feels like. This gives them a deeper appreciation for what it feels like to be moved through a cave. For example, you quickly realize that perhaps being dragged over the floor is less than ideal. Or, you learn as a patient what it feels like when your rescuers become nameless and faceless behind the glare of a dozen headlamps; next time you’re you’re a rescuer, you tend to keep in mind there’s an actual patient there and talk to them and treat them like an actual person, not a lump you’re moving through the cave.

And this leads to one of the biggest reasons: we don’t want our students to get in the habit of treating a patient like a lump in a litter. We want them to realize there’s an actual person in there.

I once did a practice rescue with a local sheriff’s department. Since it was their exercise, they set the rules. They elected to use a straw dummy as the patient. They congratulated themselves on a successful rescue at the end of the exercise. I saw a disaster. For one thing, the litter was so light, they could have probably had one person pick it up and carry it out of the cave. This may sound like a minor or even funny nit to pick. But, it can lead the Incident Commander to misjudge the crew size that may be necessary in a real rescue. (We had a cave rescue here in New York State about 20 years ago where the patient was only 300 feet into the cave. It was so arduous that we ended up having to fly in cavers from West Virginia; all the local cavers who could fit were completely exhausted.)

Because of the lightness they were practically bouncing the litter off the ceiling and walls because straw dummies don’t scream in pain when they hit rock. If they had tried to move an actual patient in that manner, they’d might have been surprised by the patient’s expressive vocabulary.

Training as one fights, or training as one rescues doesn’t necessarily mean that every scenario exactly recreates what you expect to happen. As another adage says, “no battle plan encounters the first contact with the enemy.” So you might train with a mock patient who is 180lbs and has a broken leg. And then in a real event, the patient is 240lbs, diabetic and has a broken pelvis, twisted ankle and dislocated elbow. So no, you’re not going to practice every scenario. But you’re going to practice the general concepts and understand the ideas behind them. You want an effective fighting force, you put them in the field. You have explosions, gunfire, smoke, rain, mud, etc. You don’t simply sit them in a classroom and discuss these points.

The flip side, fight as you train is important too. When the fighting or rescuing begin, you can draw upon your experience in training and will be far less panicked. I know at the few rescues I’ve been involved in, that once I’m on site, I’ve become very calm. The training clicks. You can usually tell the untrained folks at an accident because they’re either panicking or have no idea what to do. The trained folks tend to react much more calmly. Also, trained people can act with a sense of urgency that doesn’t look like panic. Untrained people often move quickly, but without a sense of purpose. Don’t confuse moving quickly with moving urgently.

And all this applies to IT. I’ve said again and again that IT departments need to exercise their disaster recovery plans. It’s great to discuss them in a meeting and have a senior manager sign off on them. It’s another thing to actual practice mock disasters. This is when you realize that “oh Shelly is out on Wednesdays afternoons and only her computer has the phone numbers of the building manager.” Or “Oh, we were sure that the batteries were in good shape, but turns out they’re getting old and we only had 1/2 the runtime we expected.” Or, as has happened too many times, “oh we thought we had good backups, until we went to restore them.”

And practicing your DR plans means you’ll be far less pressured when you execute them and as a result will make far fewer mistakes.

Today’s take-away: practice until it becomes second nature so that when you need to act for real it is second nature.

Last week one of my readers, Derek Lyons correctly called me out on some details on my post about Lock outs. Derek and I go back a long ways with a mutual interest in the space program. His background is in nuclear submarines and some of the details of operations and procedures he’s shared with me over the years have been of interest. The US nuclear submarine program is built around “procedures” and since the adoption of their SUBSAFE program, has only suffered one hull-loss and that was with the non-SUBSAFE-certified USS Scorpion.

The space program is also well known for its heavy reliance on procedures and attention to detail and safety. Out of the Apollo 13 incident, we have the famous quote, “Failure is not an option” attributed to Gene Kranz in the movie (but there’s no record of him saying it at the time.)

Anyway, his comments got me thinking about failures in general.

And I’d argue that with certain activities and at a certain level, this is true. When it comes to bringing a crew home from the Moon, or launching nuclear missiles, or performing critical surgeries, failure is not an option.

But sometimes, not only is it an option I’d say it’s almost a requirement. I was reminded of this at a small event I was asked to help be a panelist at last week. It turned out there were 3 of us panelists and just 2 students from a local program to help folks learn to code: AlbanyCanCode. The concept of agile development was brought up and the fact that agile development basically relies on failing fast and early. For software development, the concept of failing fast really only costs you time. And agile proponents will argue that in fact it saves you time and money since you find your failures much earlier meaning you spend less time going down the wrong path.

But I’m going to shift gears here to an area that’s even more near and dear to my heart: cave rescue. At an overarching, one might say strategic level, failure is not an option. We teach in the NCRC that our goal is to get the patient(s) out in as good or better shape than we found them as quickly and safely as possible. In other words, if we end up killing a patient, but get them out really quickly, that’s considered a failure; whereas if we take twice as long, but get them out alive, that’s considered a success.

But how do we do that? Where does failure come into play?

One of the first lessons I was taught by one of my mentors was to avoid “the mother of all discussions.” This lesson hit home during an incident in my Level 1 training here in New York. We had a mock patient in a Sked. Up to this point it had been walking passage through a stream with about 1″ of water. But we had hit a choke point where the main part of the ceiling came down to about 12″ above the floor passage. There was alternative route that would involve lifting the patient up several feet and then over some boulders and through some narrow and low (but not 12″ low passage) and then we’d be back to walking passage. I and two others were near the head of the litter. At this point we had placed the litter on the ground (out of the water). We scouted ahead to see how far the low passage went and noticed it went about a body length. A very short distance.

Meanwhile the rest of our party were back in the larger passage having the mother of all discussions. They were discussing whether we should could drag the litter along the floor, lift it up to go high, or perhaps even for this part, remove the patient from the litter and have them drag themselves a bit. There may have been other ideas too.

My two partners and I looked at each other, looked at the low passage, looked at the patient, shrugged our shoulders and dragged the patient through the low passage to the other side.

About 10 seconds later someone from the group having the mother of all discussions exclaimed, “where’s the patient?”

“Over here, we got him through, now can we move on?”

They crawled through and we completed the exercise.

So, our decision was a success. But what if it had been a failure. What if we realized that the patient’s nose was really 13″ higher than the floor in the 12″ passage. Simple, we’d have pulled the patient back out. Then we could have shut down the mother of all discussions and said, “we have to go high, we know for a fact the low passage won’t work.”

Failure here WAS an option and by actually TRYING something, we were able to quickly succeed or fail and move on to the next option.

Now obviously one has to use judgement here. What if the water filled passage was 14″ deep. Then no, my partners and I certainly would NOT have tried to move the patient with just the three of us. But perhaps we might have convinced the group to try.

The point is, sometimes it can often be faster and easier to actually attempt a concept than it is to discuss it to death and consider every possibility.

Time and time again I’ve seen students in our classes fall into the mother of all discussions rather than actually attempt something. If they actually attempt something they can learn very quickly if it will work or not. If it works, great, the discussion can now end and they can move on to the next challenge. If it doesn’t work, great, they’ve narrowed down their options and can discuss more intelligently about the remaining options (and then perhaps quickly iterate through those too.)

So today’s take away, is don’t be afraid of failure. Embrace it. Enjoy it. Experience it. It will lead to learning. Just make sure you understand the price of failure. Failure may be an option and is sometimes mandatory, but in other cases, the old saw is true, failure is not an option, especially if failure means the loss of life.

As I’ve mentioned, not only am I fascinated by disasters and their root causes and how we react, I’m also fascinated by how we take steps to prevent them. In my book IT Disaster Response: Lessons Learned in the Field I discuss the idea of blue-flagging on railroads. The important concepts were two-fold: 1) a method of indicating that the train should not be moved and 2) controls on who could remove that indication.

During my recent power outage, I came across something similar. It should be the featured image for this article. It’s basically an orange flag locked to a utility pole. Note the key word there, locked.

The photo doesn’t show the fact that this utility pole contained circuit breakers (I believe that’s the proper term in this context) for the overhead power lines. They had been tripped as a result of a tree further down the road taking out all three supply lines.

Close-up with tag and flag.

So let’s analyze this a bit:

The orange flag itself was VERY visible. This ensures any other crews that might be in the area that there is something they need to notice.

There is a tag with detailed information. It’s hard to see in the above photo, but it includes who tagged it, the location, and date and some other information.

What’s not clear, is it’s padlocked to the pole.

Now, to be clear, this is NOT a physical lock-out like you see on some power panels (i.e. where the padlock physically prevents the circuit-breaker from being opened or closed).

In this case, a physical lock-out would most likely have to be placed 30′ in the air at the top of the pole where it wouldn’t be easily noticed.

But that said, this served its purpose. It alerted other crews to a danger in the area and presumably can only be removed by the person who put it there. And it contains information on that person so they can be reached if there are questions.

Since power was restored within 1 hour and I didn’t hear of any reports of line worker getting electrocuted, this appears to have worked.

Today’s take-away: when you have a change from the normal state of operation, what steps can you take to ensure that others don’t try to return items to a normal state of operations without confirming things first? By the way, for a good read-up on how bad things can go when intentions about a non-standard mode of operation don’t get properly communicated, I recommend reading up on the events leading up to the Chernobyl disaster.

Procedures are important. Deviating from them can have serious consequences. Do what you can to minimize the possibility of deviations.