Life Guarding Your Data

Last week as an adult leader of my son’s Boy Scout Troop, I accompanied a dozen fine young men to a week-long summer camp. While they were working on various merit badges, I took a training course that was available to both Scouts and adult leaders – the BSA Certified Life Guard course.

There’s a reason that BSA Certified Life Guards are so well respected in the life guard community. It’s a rigorous program that’s both physically and mentally challenging. It’s considered the cream of the crop for life guard programs.

As I look back at the course, I see that there are some principles that are common to life guarding and protecting your server’s data.

Health Review

Prior to going to any camp outing, each Scout must turn in a health form that documents any medical conditions of the Scout. This allows the medical professionals to be better equipped with specific information should an emergency arise. For example, they know the medications he’s taking, conditions he has, etc.

A Server Run Book serves the same purpose in the IT world. It documents the settings on the server, the applications it supports, and the changes recently made to its configuration. Without a Run Book, diagnosing a problem is much more difficult, especially if multiple people administer the server.

Have a Plan

When things go wrong, it’s not the time to stop and think about your course of actions; it’s time to act. You need to know who’s going to do what and when. In Life Guarding it’s called a ERP, Emergency Response Plan, and it can be the difference between life and death.

The same is true for server issues. If your SQL Server goes down, you need to have a plan. You need to know who to call for what support. Know the contact information for the SAN administrator, your Windows administrator, and your boss. Know who is going to get the messaging out to the users. In short, know who will do what.

Know Your Tools

A Life Guard has many tools available to him to rescue drowning victims. He can use a pole to reach out the the person, a floating ring to toss the the victim, canoes or other boats to go get them, or a floatation tube to help in a personal rescue attempt. The Life Guard must be familiar with each of these devices before the need arises.

Likewise, a DBA has many tools at his disposal when recovering a downed server. Some are built into the SQL Server SKU, others are available through third parties. In any case, you should become fluent in their use before you need them in an emergency situation.

Keep in Shape

As a Life Guard -in-training, I had to swim, and swim, and swim. And then swim some more. And when the swimming was done, we started the training. The swimming was just a warm up.

As a DBA, it’s imperative that you keep in shape as well. Go to training. Go to conferences. Go to user group meetings. Learn the features of the new version. Know what tools are available. Incorporate the best practices into your environment. Read.

Practice

During the 30 hours of Life Guard training I received last week, almost all of it was in the form of practice. The instructor explained various techniques to us while standing on the dock and then minutes later we were in the water practicing those life saving techniques. It’s one thing to know in your head how to do something; it’s something else entirely to actually be able to do it when the pressure is on.

As DBAs we should test and practice as well. Regularly verify the integrity of your servers’ back ups. Perform some test restores to make sure that everything is happening as you wish. Don’t be caught off guard with a need to recover data only to find you can’t.

As I tell my students and clients – back ups are really all that important. It’s the ability to restore that’s important.

Know the Risks

Life Guarding is a inherently dangerous activity. You must be ready to risk your life to save another. Additionally there are legal implications to being a Life Guard.

As a Database Administrator, we have risks too. How much data can we afford to lose? How much does each minute of downtime cost our organization? The answer to these questions will help us to understand the risk and make recommendations.

Rate this:

Related

11 Responses to Life Guarding Your Data

I just came back from a missions camp (last full week of June) and we had one set of parents “forget” to disclose that their child had a severe case of Asperger’s Syndrome, probably the worst I’ve ever seen. Not knowing that, it prevented us from having a plan to deal with the issue ahead of time and caused a bunch of headaches.

If an application or a server has known issues and you “forget” to disclose, the plan to cover that application/server isn’t going to cover that issue, meaning headaches (and potential job loss) are sure to follow.

Excellent point. Hiding something about an app doesn’t mean you’ve gotten away with something. It only means that diagnosing the problems that are sure to come will be much harder, will take longer, and will likely result in a hard feelings.

Interesting metaphor. I just came to your site by chance, and it reminded me the 3 lifes me and a friend saved in the ocean, during the summer a few years ago. That tells me a lot about how things are also ‘dangerous’ for the incaut in the internet business.
Thank you.