Two data center hosting companies, one building and Hurricane Sandy

Updated Nov. 1 – As Hurricane Sandy bore down on the East Coast on Sunday, a data center
building in Manhattan lay directly in the path of the storm surge, leaving its staff to go above
and beyond to keep operations running where possible, and to recover them when it wasn’t.

This meant that hundreds of customers of Peer 1 Hosting and Internap Network Services
Corporation, located in Manhattan’s Zone A where floods took place, had the barest minimum of
downtime during and after the unprecedented storm.

“The [IT] infrastructure has held up pretty darn well so far,” said Carl Brooks, analyst at
Tier1 Research on Wednesday afternoon. “Overall, it’s been remarkable.”

Peer 1 Hosting, which has a colocation facility at the site, managed to avoid an outage when a
basement fuel tank for the building’s diesel generators flooded. The company had its own auxiliary
generator on the 17th floor, fed by a tank located on the 18th floor, but no
way to pump fuel from a truck on the ground up to that tank.

“The fuel line coming out of the truck … is four inches [wide], whereas the fuel pipe from the
basement coming from the large tanks was a three-quarter inch pipe,” said Robert Miggins, senior
vice president of business development for Peer 1. “We got the fuel really close to where it needed
to be, and all of a sudden we’re looking at these different sized delivery pipes – neither we nor
the fuel company had the proper coupling to tie

Continue Reading This Article

Enjoy this article as well as all of our content, including E-Guides, news, tips and more.

By submitting your personal information, you agree to receive emails regarding relevant products and special offers from TechTarget and its partners. You also agree that your personal information may be transferred and processed in the United States, and that you have read and agree to the Terms of Use and the Privacy Policy.

into that existing infrastructure.”

So, Peer 1’s five on-site staff began to carry fuel in small containers up the stairs to the
auxiliary generator’s tank.

They were soon joined by independent contractors, friends working as volunteers, and ultimately,
Peer 1 customers who’d arrived on the scene to try to take care of their servers – in the end, all
told, 30 people participated in the bucket brigade to keep Peer 1’s data center powered on and its
network connected.

“Customers turned up expecting to take care of themselves or take measures that they had in
mind, but instead once they found out what [our people] were working on, they rolled up their
sleeves and joined in,” said Miggins.

“They’ve been there ever since.”

This wasn’t the only crisis Peer 1 faced – with the building’s generators down, air conditioning
for one of its data center sites soon became problematic. Though there were posts on Wednesday
morning that Site 1 on the 17th floor must be shut down due to cooling issues, Miggins
said no part of the data center was powered down throughout the hurricane and its aftermath.

As of Wednesday afternoon, the bucket brigade had amassed some eight hours’ worth of fuel in the
auxiliary tank and on the ground beside it, not only catching up to the demand for more fuel but
actually buying some breathing room.

Internap makes an amazing recovery

Meanwhile, Internap, which has operations at 75 Broad Street as well as 111 8th
Street downtown, had staff at the location working around the clock at the site as well, in some
cases for two days, according to Steven A. Orchard, senior vice president of development and
operations.

On Sunday night, Internap switched over to generator power as it would do in any case of
emergency or power outage. However, a diesel fuel tank and pump were located in the basement of the
building, which soon began to flood. Salt water also eventually breached the fuel tank through a
ventilation pipe which ran outside the building into the street.

Meanwhile, by Monday night, Internap’s staff found themselves trapped in a harrowing scene – on
the 14th floor of a building where flood waters were rising into the lobby, reeking of diesel from
the basement tank and gasoline from flooded-out cars.

At this point, customers were advised that plans were being made about how to evacuate the
building in case of a fire, although nobody did evacuate, Orchard said.

Internap’s generators are up on the mezzanine level of the building, above the flood waters. And
a header tank with about 1,500 gallons of fuel was up there, enough to last until about 11 a.m.
Tuesday. Then, and only then, did Internap lose power to the site.

Internap’s primary diesel fuel vendor’s supplies were being exhausted by a number of hospitals
also on its route, so Internap looked to a secondary vendor, which trucked fuel into the city out
of Baltimore, arriving late Tuesday afternoon.

But the recovery was far from assured at that point. Two other things were needed:
couplings and piping to be able to get the fuel out of the truck, and a pump to get it up to the
14th floor.

Here Internap turned to another of its vendors, an HVAC company, whose workers fabricated custom
couplings and pipe fittings, attaching the truck to a pipe that ran up the side of the building, in
through an opening in the wall, and into the header tank.

Tuesday evening, the infrastructure was turned back on. After an unprecedented storm and
flooding, Internap’s customers had seen just a nine-hour outage.

“I don’t think people outside the city have a clear understanding of exactly how devastating it
was and how frightening for the team that was there,” Orchard said. “I can’t say enough about
them.”

Beth Pariseau is a senior news writer for SearchCloudComputing.com and
SearchServerVirtualization.com. Write to her at bpariseau@techtarget.com or follow @PariseauTT on
Twitter.