Coverage

After Sandy, Manual Labor Keeps Cloud Services Running

Arik HesseldahlNovember 1, 2012 at 5:49 pm PT

The whole point about cloud computing services that live on servers
that customers never see, never touch, never think about is exactly
that. The physical infrastructure is out of sight, out of mind, and
customers focus on running their applications and using the services
they provide.

But when disaster hits, when power fails followed by backup power
failing, things get serious and physical rather quickly. That has been
the case with Squarespace, an upstart Web site creation and hosting company based here in New York.

It has hosted its infrastructure with Peer1, one of a handful of companies affected by the flooding at 75 Broad Street. The short version of the story is that after backup generators kicked in, flooding overwhelmed the fuel pump.

CEO Anthony Casalena lives nearby in Soho and headed down to 75 Broad
to see if he could lend a hand. Within hours he was pitching in with
the staff of Peer1, manually carrying fuel to the generators,
bucket-brigade style up 17 flights of stairs to keep them running. Staff
from another company, Fog Creek Software, was also involved. The picture is of the “bucket brigade” from last night.

The result is that they managed to keep the systems running by
manually feeding the generators while searching desperately for another
way to keep a steady flow of fuel until reliable commercial power is
turned back on. The best guesses we’re hearing from ConEdison is that
this won’t be until Friday or Saturday at the earliest, maybe longer. The entire saga has been chronicled on a status blog with updates like this from 11 pm ET yesterday:

The midnight bucket brigade has begun (photo below).

When tonight’s brigade is done, we estimate we have a conservative 10
hours of capacity, allowing our 8am shift to begin tomorrow with low
stress and also a switch to 120 minute refill sessions every 8 hours,
versus yesterday’s continuous huffing of fuel up the stairs. Yeah, we’re
getting good at this.

The building has had crews pumping water out of the basement all day.
Unfortunately, we have no ETA on the repairing of the basement fuel
delivery systems until the building’s crews get down there. However,
both the building and PEER1 have been putting together separate
contingency fuel pumping systems. We hope one of these will be online in
the next 24-36 hours so that fuel refills will involve very few people
and simply take place via a truck at street level.