Coverage

Hurricane Sandy has been a nightmare. It destroyed homes, knocked out
power, flooded the New York City Subway, and wiped out about a quarter
of the cellphone towers between Virginia and Massachusetts. And now we
know that it also cut off about 10 percent of the computer networks in
Manhattan.

That’s according to Renesys, an internet monitoring company that has also tracked internet outages in Egypt and Syria.

Networks went down when the local utility company, Con Edison, cut
power Monday night to protect equipment from the storm surge brought on
by Sandy. That knocked many of the networks offline, including those
hosted at Datagram, a popular data center in lower Manhattan. Others
managed to stay online for much of the past three days, but for some, it
was a very rough time.

“The 90 percent that survive are in data centers, running on
generator power supplied by engineers who do not sleep much,” Renesys
chief technology officer James Cowie wrote on a company blog.

We spoke with one of those companies running on backup generators,
Peer 1 Hosting, which formed a bucket brigade to haul diesel fuel up 17
flights of stairs to its data center when basement fuel pumps got
flooded and failed.

Peer 1 warned customers it was going to have to turn off its data
center when reserve fuel ran out, but it tried to keep things going as
long as possible. When customers showed up Tuesday and saw five staffers
trying to haul 50 gallons of diesel fuel per hour up 17 flights of
stairs, they decided to help out.

“Their solution is their livelihood. So they wanted to pitch in and
help and keep it up as much as possible,” says Ryan Murphey, vice
president of data center operations at Peer 1.

One of those customers, Squarespace, has been describing its Sisyphean efforts to help keep Peer 1′s generators running on this blog.