Disasters could take down half of data centers worldwide. There’s a way to protect the data

The flood has gotten everywhere. Both the ground floor and the basement with the generators’ fuel were under water. It was a disaster. “We really didn’t expect the storm to have a significant effect on the data center,” Peer 1’s vice president Ryan Murphey recounted a year later. His company had not prepared for the hurricane Sandy that was about to hit New York. Only two employees were left in the data center at the time. He added: “At the worst, we thought the facility might lose utility power, and Jeff and Mike would need to switch to the building’s backup generators.”

Their servers, located in the building’s second floor, survived the storm intact.

However, the water flooded the underground, where fuel reserves were located. As well as pumps that were to carry that fuel up to the generators on the seventeenth floor. The storm disrupted not only the power coming from the grid, but also the data center’s backup system.

The generators were okay, but they were running out of fuel fast. Their fuel consumption was too high to be covered just by a small backup fuel tank on the generators’ floor.

The disaster created one of the most engaging data center stories of the recent past.

‘Bucket Brigade’ took on hundreds of stairs

The two engineers thought of contacting other employees. All those people then created a chain taking buckets of fuel up the stairs to the seventeenth floor. Their activity sparked so much interest that the data center’s customers with their employees and even passers-by started joining in. “People were tired of being helpless against the storm, they were thrilled they could do something with a purpose,” said one of the volunteers. After the storm cleared out, their actions piqued the interest of media and ‘the bucket brigade’ as they were soon dubbed, even got a documentary made about them.

Thanks to their cooperation, the generators survived the storm and its aftermath without a failure. After a few days, the tired but proud and happy team of volunteers was replaced with backup pumps and fuel hoses. The data center has survived the storm without any of its customers losing access to their services.

The hurricane did take its toll on the facility, though. “Looking back, it was a very rough recovery and one that I hope to never experience again,” Murphey admitted.

Their company and its clients were incredibly lucky. The data center had no complex system of coping with such a huge failure. At least four other data centers in the same part of New York were not so lucky and failed.

A half of data centers has suffered at least 5 failures due to natural disasters over the last ten years

Whether a data center is worried about floods, fires or earthquakes, you want your data to be absolutely safe and always available.

But threats can be considered way ahead – and they often are, especially when choosing a location for a new data center. Services offering clients to save their data, infrastructure and processing power into more locations at once are growing more and more common. When one of the data center suffers even a short-term failure, its clients won’t notice anything. The cloud can be ordered to switch them over to the second data center.

Services like these are usually called Site Recovery. Master DC offers this service to its cloud customers as well. Client’s infrastructure is replicated in its entirety to a second data center and is guaranteed the same amount of performance as in the primary data center. The entire infrastructure and the data is also located in facility more than 200 kilometres away, where they are sure to be safe from a local disaster.

Thanks to a regular replication, your infrastructure and data is instantly ready even when one facility fails. And it’s possible to store older backups, in case you want to go back to them – so if an operation system fails, you can go back to a version that works. And you can set the timing of replications yourselves.

Unprepared data centers will not avoid failures when disasters strike. When choosing a provider for your cloud, pick a partner that is ready for them and that can make them seem trivial.