Category Archives: Management

Post navigation

In a previous post, Social Deconstruction I reflected on a barrier that had been put up on a Thursday, and by Sunday, completely bypassed. I had recent cause to revisit that area again recently and

Barrier bypassed

as you can see, an actual, real gate has been put into the fence. The power of the crowd basically overruled the original intent of the landowner.

Of course, this could have been done from day one.

This is true in the IT world. How often has the security department come and said, “we’re implementing this new security policy” with little input from actual users and are surprised when users get frustrated and try to bypass the new security feature. I had this happen at a client of mine. In the case of the fence above, people bypassed the security the fence builders wanted (presumably to reduce liability), and by doing so, increased their chance of getting hurt (and ironically, presumably increasing liability).

One of the security features that I think annoys most of us are passwords, or more accurately arcane password requirements. For example, some systems require a certain amount of complexity, but don’t necessarily tell you what the rules for complexity actually are! Yes, I’ve had that happen. Turns out they required special characters, but, only a specific subset of special characters and the ones I tried weren’t on that subset.

Now a minimum password length, makes sense. A one character password can be cracked by anyone. But, what about short maximum password lengths? Yes, perhaps that was a good idea when memory and storage were scarce (ok even then, not a great idea) but not so much these days. Yet, I know at least one system where your password has to be between 8 and 14 characters.

Another annoyance is the “must change every N days” where often N is something like 90 (though I’ve seen even lower). What does this mean? Folks end up with passwords like: Secur3Passwrd$1, Secur3Passwrd$2, Secur3Passwrd$3, etc.

Truth is, many of the so called password rules, actually encourage us to create lousy password, and so we repeat stuff, or write it down or take other steps that make it easier for to use them, but also as a byproduct weaken passwords.

The National Institute of Standards and Technology recently released an updated set of guidelines: NIST 800-63B that discuss good password requirements (note I have NOT read the entire document, just large portions of it). Spycloud has a decent review here: New NIST Guidelines Acknowledge We’re Only Human. I’m not going to recap the recap here, but I will add what I generally do:

I use a password manager. You can read reviews for finding one that best meets your needs. Personally, I use one that does NOT have storage on the cloud. While in theory they’re encrypted and secure, I get paranoid. (Yes, I do recognize if someone compromises my desktop, they can get access to my local password manager. But on the other hand, if they get access to my desktop, they can probably just install a keyboard logger and I’m hosed anyway.)

I use a different password, automatically created by the above password manager for nearly every site of system I log into. This ends up meeting most (but not all) of the NIST suggestions (they’re certainly NOT easy to remember, but they don’t have dictionary words, can be as long as I need, most likely are NOT in a previous breech, etc.)

Note, I said most, not all. There’s a few places I used passwords I can remember. These are systems I interact with on a daily or near daily basis, such as my desktop, AND the password manager itself. There would be no point to have a password manager if I couldn’t log into it, or if the password were so simple anyone could guess it.

So, I make sure these passwords are easy to remember, but extremely hard to guess. (For example, they do NOT include the name of my first dog, my mother’s maiden name, etc.)

In conclusion, if you’re in charge of security, make it usable, or else people WILL try to bypass it, simply to get the job done. And, remember, you’re always in charge of your own security, so make it usable, but secure.

I was going through my old drafts and found this post I had started to write earlier this year but never finished. Actually it appears I meant this to be part of White (K)nights but I cut it out to make that post more readable.

During my media interactions I was asked multiple times to comment on Elon Musk and once or twice on his submarine. I tried to keep my comments fairly neutral, but the truth is, I and some of my fellow trained cave rescuers were pretty bothered by Musk’s attempted involvement. I got into at least one online debate about how the people in charge obviously were clueless and that Musk’s solution of a submarine was a brilliant idea.

It wasn’t and I figured I’d address some of my concerns. Please note as with all situations like this, I was not directly involved, so I’m going on publicly available facts and my training as a cave rescue person and a cave rescue instructor. I am also not in any way speaking on behalf of the National Cave Rescue Commission or the NSS.

Now let’s discuss the device itself:

It almost certainly would not have fit. By all accounts, the tightest pinch was 15″ and hard to navigate. Anyone who has moved through a cave knows that even larger passages can be hard to navigate. Locally we have a cave that has a pinch that’s probably close to 15″, but that is at the bottom of a body sized V-shaped passage. Unless you can bend in the middle, you will not fit through it. A cylinder like Musk designed, would not fit. I don’t know the passages in the Thai cave, but odds are there is more than one passage where flexibility is important.

It also, in many ways was superbly dangerous. Once sealed into the tube, there would be no easy way to monitor the patient’s vitals. And if the tube had started to leak (cave environments can be extremely destructive, even to metal objects), there appears there would have been no recourse except to keep swimming and hoping to get to an air filled chamber quickly enough and that was large enough to debug the issue.

In addition, if the patients were not sedated, I’d have to imagine that being sealed into such a tube, even with lights for 20-40 minutes at a time would have been sheer terror. As it is, the kids were in fact apparently heavily sedated (a fact that some of us still find a bit surprising, even though very understandable), and yet at least one started to come out of sedation while in a water passage. Without being able to directly monitor the vitals of the patient, who knows what would have happened.

There’s probably other issues I could come up with. But let me end with this one. Rarely if ever do you want to beta-test or heck even alpha-test, which is what this would have been, a brand new design in a life or death situation when there are alternatives.

Like our White Knights, we want our brilliant tech solutions, but often we’re better off adapting what we’ve done in the past. In cave rescue we try to teach our students a “bag of tricks” that they can adapt to each particular rescue. Foe example, there is no single rigging solution that will work for every rescue. How I might rig a drop in Fantastic in Ellison’s might be very different from how I’d rig a drop here in New York. How I package a patient for movement here may be different than in a Puerto Rican cave. And honestly I’ve seen a lot of high-tech equipment get suggested for cave rescue that simply doesn’t work well in a cave environment and we often go back to the simple proven stuff.

I will add a tease, to perhaps a future blog post, of a mock rescue rescue where a high-tech approach failed after several hours of trying, and the low-tech solution solved the problem.

There’s a military aphorism “Train as you fight, fight as you train.” I was recently reminded of this by a friend mine and a reader of my blog. We’ve shared a mutual interest in the space program for decades. He mentioned this last week (though I can’t seem to find the post) in response to something I wrote and it got me thinking.

When we teach cave rescue, we almost always use a real patient in the litter. There’s a couple of reasons for this. For one, it ipso facto recreates the actual mass and weight distribution of a real patient. Now, there are training dummies that are similar in weight and mass, but they can be a pain in the neck. For one thing, ever try to move an inert body? That’s what a training dummy can be like. Sure it’s great once it’s IN the litter, but getting it into position deep inside a cave can be almost impossible.

For another, it gives our students a chance to experience what being a patient feels like. This gives them a deeper appreciation for what it feels like to be moved through a cave. For example, you quickly realize that perhaps being dragged over the floor is less than ideal. Or, you learn as a patient what it feels like when your rescuers become nameless and faceless behind the glare of a dozen headlamps; next time you’re you’re a rescuer, you tend to keep in mind there’s an actual patient there and talk to them and treat them like an actual person, not a lump you’re moving through the cave.

And this leads to one of the biggest reasons: we don’t want our students to get in the habit of treating a patient like a lump in a litter. We want them to realize there’s an actual person in there.

I once did a practice rescue with a local sheriff’s department. Since it was their exercise, they set the rules. They elected to use a straw dummy as the patient. They congratulated themselves on a successful rescue at the end of the exercise. I saw a disaster. For one thing, the litter was so light, they could have probably had one person pick it up and carry it out of the cave. This may sound like a minor or even funny nit to pick. But, it can lead the Incident Commander to misjudge the crew size that may be necessary in a real rescue. (We had a cave rescue here in New York State about 20 years ago where the patient was only 300 feet into the cave. It was so arduous that we ended up having to fly in cavers from West Virginia; all the local cavers who could fit were completely exhausted.)

Because of the lightness they were practically bouncing the litter off the ceiling and walls because straw dummies don’t scream in pain when they hit rock. If they had tried to move an actual patient in that manner, they’d might have been surprised by the patient’s expressive vocabulary.

Training as one fights, or training as one rescues doesn’t necessarily mean that every scenario exactly recreates what you expect to happen. As another adage says, “no battle plan encounters the first contact with the enemy.” So you might train with a mock patient who is 180lbs and has a broken leg. And then in a real event, the patient is 240lbs, diabetic and has a broken pelvis, twisted ankle and dislocated elbow. So no, you’re not going to practice every scenario. But you’re going to practice the general concepts and understand the ideas behind them. You want an effective fighting force, you put them in the field. You have explosions, gunfire, smoke, rain, mud, etc. You don’t simply sit them in a classroom and discuss these points.

The flip side, fight as you train is important too. When the fighting or rescuing begin, you can draw upon your experience in training and will be far less panicked. I know at the few rescues I’ve been involved in, that once I’m on site, I’ve become very calm. The training clicks. You can usually tell the untrained folks at an accident because they’re either panicking or have no idea what to do. The trained folks tend to react much more calmly. Also, trained people can act with a sense of urgency that doesn’t look like panic. Untrained people often move quickly, but without a sense of purpose. Don’t confuse moving quickly with moving urgently.

And all this applies to IT. I’ve said again and again that IT departments need to exercise their disaster recovery plans. It’s great to discuss them in a meeting and have a senior manager sign off on them. It’s another thing to actual practice mock disasters. This is when you realize that “oh Shelly is out on Wednesdays afternoons and only her computer has the phone numbers of the building manager.” Or “Oh, we were sure that the batteries were in good shape, but turns out they’re getting old and we only had 1/2 the runtime we expected.” Or, as has happened too many times, “oh we thought we had good backups, until we went to restore them.”

And practicing your DR plans means you’ll be far less pressured when you execute them and as a result will make far fewer mistakes.

Today’s take-away: practice until it becomes second nature so that when you need to act for real it is second nature.

Last week one of my readers, Derek Lyons correctly called me out on some details on my post about Lock outs. Derek and I go back a long ways with a mutual interest in the space program. His background is in nuclear submarines and some of the details of operations and procedures he’s shared with me over the years have been of interest. The US nuclear submarine program is built around “procedures” and since the adoption of their SUBSAFE program, has only suffered one hull-loss and that was with the non-SUBSAFE-certified USS Scorpion.

The space program is also well known for its heavy reliance on procedures and attention to detail and safety. Out of the Apollo 13 incident, we have the famous quote, “Failure is not an option” attributed to Gene Kranz in the movie (but there’s no record of him saying it at the time.)

Anyway, his comments got me thinking about failures in general.

And I’d argue that with certain activities and at a certain level, this is true. When it comes to bringing a crew home from the Moon, or launching nuclear missiles, or performing critical surgeries, failure is not an option.

But sometimes, not only is it an option I’d say it’s almost a requirement. I was reminded of this at a small event I was asked to help be a panelist at last week. It turned out there were 3 of us panelists and just 2 students from a local program to help folks learn to code: AlbanyCanCode. The concept of agile development was brought up and the fact that agile development basically relies on failing fast and early. For software development, the concept of failing fast really only costs you time. And agile proponents will argue that in fact it saves you time and money since you find your failures much earlier meaning you spend less time going down the wrong path.

But I’m going to shift gears here to an area that’s even more near and dear to my heart: cave rescue. At an overarching, one might say strategic level, failure is not an option. We teach in the NCRC that our goal is to get the patient(s) out in as good or better shape than we found them as quickly and safely as possible. In other words, if we end up killing a patient, but get them out really quickly, that’s considered a failure; whereas if we take twice as long, but get them out alive, that’s considered a success.

But how do we do that? Where does failure come into play?

One of the first lessons I was taught by one of my mentors was to avoid “the mother of all discussions.” This lesson hit home during an incident in my Level 1 training here in New York. We had a mock patient in a Sked. Up to this point it had been walking passage through a stream with about 1″ of water. But we had hit a choke point where the main part of the ceiling came down to about 12″ above the floor passage. There was alternative route that would involve lifting the patient up several feet and then over some boulders and through some narrow and low (but not 12″ low passage) and then we’d be back to walking passage. I and two others were near the head of the litter. At this point we had placed the litter on the ground (out of the water). We scouted ahead to see how far the low passage went and noticed it went about a body length. A very short distance.

Meanwhile the rest of our party were back in the larger passage having the mother of all discussions. They were discussing whether we should could drag the litter along the floor, lift it up to go high, or perhaps even for this part, remove the patient from the litter and have them drag themselves a bit. There may have been other ideas too.

My two partners and I looked at each other, looked at the low passage, looked at the patient, shrugged our shoulders and dragged the patient through the low passage to the other side.

About 10 seconds later someone from the group having the mother of all discussions exclaimed, “where’s the patient?”

“Over here, we got him through, now can we move on?”

They crawled through and we completed the exercise.

So, our decision was a success. But what if it had been a failure. What if we realized that the patient’s nose was really 13″ higher than the floor in the 12″ passage. Simple, we’d have pulled the patient back out. Then we could have shut down the mother of all discussions and said, “we have to go high, we know for a fact the low passage won’t work.”

Failure here WAS an option and by actually TRYING something, we were able to quickly succeed or fail and move on to the next option.

Now obviously one has to use judgement here. What if the water filled passage was 14″ deep. Then no, my partners and I certainly would NOT have tried to move the patient with just the three of us. But perhaps we might have convinced the group to try.

The point is, sometimes it can often be faster and easier to actually attempt a concept than it is to discuss it to death and consider every possibility.

Time and time again I’ve seen students in our classes fall into the mother of all discussions rather than actually attempt something. If they actually attempt something they can learn very quickly if it will work or not. If it works, great, the discussion can now end and they can move on to the next challenge. If it doesn’t work, great, they’ve narrowed down their options and can discuss more intelligently about the remaining options (and then perhaps quickly iterate through those too.)

So today’s take away, is don’t be afraid of failure. Embrace it. Enjoy it. Experience it. It will lead to learning. Just make sure you understand the price of failure. Failure may be an option and is sometimes mandatory, but in other cases, the old saw is true, failure is not an option, especially if failure means the loss of life.

I apologize for skipping two weeks of blog posts, but I was a bit busy; for about 11 days my family and I were visiting Europe for the first time. It was a wonderful trip. It started with a trip to Manchester UK for a SQL Saturday event.

I had sort of forgotten exactly how much further north we were until it dawned on me how early dawn was. Actually we had noticed the night before as we walked back from the amazingly wonderful speakers’ dinner how light it was despite how late it was. When I woke up at around 4:30 AM (a bit of jetlag there) I noticed despite the blackout curtains how bright it was around their edges. I later looked it up, and it appears that technically it never reached “night” there, but simply astronomical twilight.

Ever since seeing the movie “White Nights” my wife has always wanted to experience the white nights of Russia. This wasn’t that, but it was close.

This trip followed up on the heels of the amazingly successful Thai Cave Rescue that I had previously commented on. As long term readers know, I’m a caver who also teaches cave rescue and has a role as the Northeast Coordinator of the National Cave Rescue Commission. During the 18 day saga, I and others were called upon by various media outlets to give our insight and perspective. I was fortunate, I only did a little under a dozen media events. Our National Coordinator, Anmar Mirza did well over 100, and most of those in about a 5 day period. A link to one of my media events is here: The Takeaway.

I don’t want to talk about the operation itself, but I want to talk about White Knights. We love our White Knights: the term often refers to a character who will ride into town and single-handedly solve the town’s problems. The truth is, white knights rarely if ever exist and that most problems require a lot more effort to solve.

We’ve seen this in politics, and we saw this with this cave rescue. Let me start by saying I think the work Elon Musk has done with SpaceX is amazing. SpaceX has in fact single-handedly revolutionized the space launch market.

It was perhaps inevitable that Musk’s name would show up in relation to this cave rescue. Musk has previously gotten attention for attempting to help with the power outage crisis in Puerto Rico and now his vow to help the people of Flint (both by the way I think worthy causes and I wish him and more importantly the people he’s trying to help, well).

But here’s the thing, a cave rescue isn’t solved by a white knight. It’s solved by a lot of effort and planning with a lot of people with a variety of skills and experience. There’s rarely a magic breakthrough that magically makes things easier.

And I’ll be blunt: his “submarine” idea, while interesting, was at best a PR distraction and at worst, possibly caused problems.

“But Greg, he was trying to help, how could this make things worse?” I actually disengaged from an online debate with some Musk fanbois who couldn’t see why Musk’s offer was problematic. To them, he was the white knight that could never do wrong.

Here’s the thing: I know for a fact that several of us, myself included, had to take part of our allotted airtime or written coverage to address why Musk’s idea probably wouldn’t work. This meant less time or room for useful information to be passed on to the audience. Part of my role as regional coordinator is to educate people about cave rescue, and I can’t do this effectively when I’m asked to discuss distractions.

“But so what, that didn’t impact the rescue.” No, it didn’t. But, it appears from the Twitter fights I’ve seen, and other information, that at least some resources on the ground were tasked to deal with Musk. This does mean that people had to spend time dealing with both Musk and the publicity. This means those resources couldn’t be spent elsewhere. At least one report from Musk (which honestly I question) suggests he actually entered the cave during the rescue operations. This means that resources had to be spent on assuring his safety and possibly prevented another person who could have provided help in other ways (even if it was simply acting as a sherpa) from entering.

And apparently, there’s now a useless “submarine” sitting outside the cave. I’ll leave discussion of why I had problems with the submarine itself for another post.

But here’s one final reason I have problem with Musk bringing so much attention to himself and his idea: It could have lead to second guessing.

Let’s be clear: even the cave divers themselves felt that they would most likely lose some of the kids; this was exactly how dangerous the rescue was. This is coming from the folks who best knew the cave and best understand the risks and issues. Some of the best cave divers in the world, with rescue experience, who were on-site, thought that some kids would die in the attempt to rescue them. And, if reports are true, they were aware of Musk’s offer and obviously rejected it (and in fact one suggested later that Musk do something anatomically impossible with it.)

Had the rescuers worst fears come true, Musk fan bois would have second guessed every decision. In other words, people would have put more faith in their favorite white knight, who had zero practical experience in the ongoing operations , than they would have in the very people who were there and actively involved. I saw the comments before and during the operations from his fans and all of them were upset that their favorite white knight wasn’t being called in to save the day. I can only imagine how bad it would have been had something tragic occurred.

This is why I’m against white knights. They rarely if ever solve the problem, and worse when they do ride into town, they take time and energy away from those who are actually working on the problems. Leave the white knights on a chess board.

As I’m writing this, word has rocketed around the world that the 12 soccer players and their coach have been safely rescued from Tham Luang cave. We are awaiting word that all the rescuers themselves, including one of the doctors that had spent time with the boys since they were found, are still on their way out.

Unfortunately, one former Thai SEAL diver, Saman Kunan, who had rejoined his former teammates to help in the rescue, lost his life. This tragic outcome should not be forgotten, nor should it cast too large of a shadow on the amazing success.

What I want to talk about though is not the cave or the rescue operations, but the decision making progress. The title for this post comes from Narongsak Osottanakorn’s statement several days ago when they began the evacuation operations.

The term D-Day actually predates the famous Normandy landings that everyone associates it with. However, success of the Normandy landings and their importance in the ultimate outcome of WWII has forever cemented that phrase in history.

One of the hardest parts of any large scale operation like this is making the decision on whether to act. During the Apollo Program, they called them GO/NO GO decisions. Famously you can see this in the movie Apollo 13 where Gene Kranz goes around the room asking for a Go/No Go for launch. (it was pointed in a Tindellgram out before the Apollo 11 landing, that the call after the Eagle landed should be changed to Stay/No Stay – so there was no confusion on if they were “go to stay” or “go to leave”.)

While I’ve never been Flight Commander for a lunar mission, nor a Supreme Allied Commander for a European invasion, I have had to make life or death decisions on much smaller operations. A huge issue is not knowing the outcome. It’s like walking into a casino. If you knew you were always going to win, it would be an easy decision on how to bet. But obviously that’s not possible. The best you can do is gather as much information as you can, gather the best people you can around you, trust them and then make the decision.

What compounds the decision making progress in many cases, and especially in cave rescue is the lack of communication and lack of information. It can be very frustrating to send rescuers into the cave and not know, sometimes for hours, what is going on. Compound this with what is sometimes intense media scrutiny (which was certainly present here with the entire world watching), and one can feel compelled to rush the decision making progress. It is hard, but generally necessary to resist this. In an incident I’m familiar with, I recall a photograph of the cave rescue expert advising rescue operations, standing in the rain, near the cave entrance waiting for the waters to come down so they could send search teams in. Social media was blowing up with comments like, “they need to get divers in there now!” “Why aren’t the authorities doing anything?” The fact is, the authorities were doing exactly what the cave rescue expert recommended; waiting for it to be safe enough to act. Once the waters came down, they could send people and find the trapped cavers.

The incident in Thailand is a perfect example of the confluence of these factors:

There was media pressure from around the world with people were asking why they were taking so long to begin rescuing the boys and once they did start to rescue them, why it took them three days. Offers and suggestions flowed in from around the world and varied from the absurd (one suggestion we received at the NCRC was the use of dolphins) to the unfortunately impractical (let’s just say Mr. Musk wasn’t the only one, nor the first, to suggest some sort of submarine or sealed bag).

There was always a lack of enough information. Even after the boys had been found, it could take hours to get information to the surface, or from the surface back to the players. This hinders the decision making process.

Finally of course are the unknowns:

When is the rain coming?

How much rain?

How will the boys react to being submerged?

What can they eat in their condition?

And finally, there is, in the back of the minds of folks making the decisions the fact that if the outcome turned tragic, everyone will second guess them.

Narongsak Osottanakorn and others had to weigh all the above with all the facts that they had, and the knowledge that they couldn’t have as much information as they might want and make life-impacting decisions. For this I have a great deal of respect for them and don’t envy them.

Fortunately, in this case, the decisions led to a successful outcome which is a huge relief to the families and the world.

For any operation, especially complex ones, such as this rescue, a moon landing or an invasion of the beaches of Normandy, the planning and decision making process is critically important and often over shadowed by the folks executing the operation. As important as Neil Armstrong, Buzz Aldrin and Michael Collins (who all to often gets overlooked, despite writing one of the better autobiographies of the Apollo program) were to Apollo 11, without the support of Gene Kranz, Steve Bales, and hundreds of others on the ground, they would have very likely had to abort their landing.

So, let’s not forget the people behind the scenes making the decisions.

“When does a cave rescue become a recovery?’ That was the question a friend of mine asked me online about a week ago. This was before the boys and their coach had been found in the Thai cave.

Before I continue, let me add a huge caveat: this is an ongoing dynamic situation and many of the details I mention here may already be based on inaccurate or outdated information. But that’s also part of the point I ultimately hope to make: plans have to evolve as more data is gathered.

My somewhat flippant answer was “when they’re dead.” This is a bit of dark humor answer but there was actually some reasoning behind it. Before I go on, let me say that at that point I actually still had a lot of hope and reason to believe they were still alive. I’m very glad to find that they were in fact found alive and relatively safe.

There’s a truth about cave rescue: caves are literally a black-hole of information. Until you find the people you’re searching for, you have very little information. Sometimes it may be as little as, “They went into this cave and haven’t come out yet.” (Actually sometimes it can be even less than that, “We think they went into one of these caves but we’re not even sure about that.”)

So when it comes to rescue, two of the items we try to teach students when teaching cave rescue is to look for clues, and to try to establish communications. A clue might be a footprint or a food wrapper. It might be the smell of a sweaty caver wafting in a certain direction. A clue might be the sound of someone calling for help. And the ultimate clue of course is the caver themselves. But there are other clues we might look for: what equipment do we think they have? What experience do they have? What is the characteristics of the cave? These can all drive how we search and what decisions we make.

Going back to the Thai cave situation, based on the media reports (which should always be taken with a huge grain of salt) it appeared that the coach and boys probably knew enough to get above the flood level and that the cave temps were in the 80s (Fahrenheit). These are two reasons I was hopeful. Honestly, had they not gotten above the flood zone, almost certainly we’d be talking about a tragedy instead. Had the cave been a typical northeast cave where the temps are in the 40s (F) I would have had a lot less hope.

Given the above details then, it was reasonable to believe the boys were still alive and to continue to treat the situation as a search and eventually rescue situation. And fortunately, that’s the way it has turned out. What happens next is still open for speculation, but I’ll say don’t be surprised if they bring in gear and people and bivouac in place for weeks or even months until the water levels come down.

During the search process, apparently a lot of phone lines were laid into parts of the cave so that easier communications could be made with the surface. Now that they have found the cavers, I’d be shocked if some sort of realtime communications is not setup in short order. This will allow he incident commander to make better informed decisions and to be able to get the most accurate and up to date data.

So, let me relate this to IT and disasters. Typically a disaster will start with, “the server has crashed” or something similar. We have an idea of the problem, but again, we’re really in a black-hole of information at that moment. Did the server crash because a hard drive failed, or because someone kicked the power cord or something else?

The first thing we need to do is to get more information. And we may need to establish communications. We often take that for granted, but the truth is, often when a major disaster occurs, the first thing to go is good communications. Imagine that the crashed server is in a datacenter across the country. How can you find out what’s going on? Perhaps you call for hands on support. But what if the reason the server has crashed is because the datacenter is on fire? You may not be able to reach anyone! You might need to call a friend in the same city and have them go over there. Or you might even turn on the news to see if there’s anything on worth noting.

But the point is, you can’t react until you have more information. Once you start to have information, you can start to develop a reaction plan. But let’s take the above situation and imagine that you find your datacenter has in fact burned down. You might start to panic and think you need to order a new server. You start to call up your CFO to ask her to let you buy some new hardware when suddenly you get a call from your tech in the remote. They tell you, “Yeah, the building burned down, but we got real lucky and our server was in an area that was undamaged and I’ve got it in the trunk of my car, what do you want me to do with it?”

Now your previous data has been invalidated and you have new information and have to develop a new plan.

This is the situation in Thailand right now. They’re continually getting new information and updating their plans as they go. And this is the way you need to handle you disasters, establish communications, gather data and create a plan and update your plan as the data changes. And don’t give up hope until you absolutely have to.