Q&A: Data Center Expert Chris Wade on Disaster Preparedness, Planning

In the wake of Hurricane Sandy, those facility managers fortunate enough to not be affected are checking emergency preparedness plans, discussing contingency options for similar situations and testing their facilities and staffs to see what works in reality and what only sounds good in theory.

In this Q&A, Chris Wade, principal, Resilient Critical Facility Solutions, offers advice on how to prepare a data center for an event like Sandy and how to make sure you can keep operating when it happens.

Q: What sort of preparation can you do in a data center for an event like Sandy?

A: You can’t wait ’til the sky is falling to start planning for man-made or natural disaster events. From a data center perspective, you must have a pre-defined disaster preparedness plan of action for events such as tornadoes, hurricanes, loss of power, fire, loss of communications, etc. Your business continuity plan should include emergency action plans and disaster recovery plans for the data center.

It is important that the proper steps are taken to ensure that the data center infrastructure support systems will stay up and running during an event such as Hurricane Sandy. It is crucial to ensure that your disaster recovery and secondary sites are available and ready to be on-line when needed. If you are using a hosting provider for your secondary or disaster recovery site, it is also important to validate that they have an adequate emergency action plan in place for disaster recovery. Many times, these types of locations are dark sites with minimal staffing. Make sure your disaster recovery plan includes a plan for ramping up the staffing at these sites to support your operations.

Here are some essential steps required to ensure a data center remains on-line and operational:

1. Ensure you can manage the load.

• Test all backup power systems (imitate a total power outage) and ensure the generators and the transfer switchgear operate properly. I have found that for most events, testing out emergency systems is the first priority. A best practice is to start with a power failure checklist. Always make sure the power system has been tested and that you can fail over to generators (or an alternate feed) for continuous operations at the facility.

• Contact your generator maintenance and service vendor to have a generator technician on site.

• Make sure fuel tanks are full, levels verified, and backup fuel vendors have been placed on standby in the event of extended power interruption. Generators eat up a lot of fuel when they are operating, and you need to know the generator burn rate (gallons per hour of fuel burned) to determine generator runtime so you will know when to call in fuel reserves.

• The emergency fuel provider should be notified immediately that you may require additional fuel, and it will be delivered within 24 hours of a call for service. It is a good practice to have a pre-defined agreement for emergency service with your fuel provider. (You won’t be the only one looking for fuel in an event of this magnitude).

• Many data centers have on-site fuel storage tanks that can provide fuel for an extended period (some can run on generator for 72 hours or more without refueling).

2. Have sufficient resources onsite and staged to operate without utility power for weeks if needed

• Additional data center engineers and technical support staff should be on-site throughout the duration of the storm to monitor operations and ensure infrastructure availability.

• Ensure your list of employee phone numbers is regularly updated and the required leadership or department head each has a copy.

• Make sure there is adequate food and water available for onsite staff in case the storm’s duration is extended or additional supplies are unavailable.