Coverage

Diesel ‘Bucket Brigade’ Keeps Peer 1 Online at 75 Broad

Teams
from Squarespace fill buckets with diesel fuel to haul them up 17
stories to the generator keeping the data center online. Staff from Peer
1, Squarespace and Fog Creek Software have formed this unusual Internet
bucket brigade. (Photo via Squarespace)

A determined team of employees of Peer 1 Hosting, blog host Squarespace and Fog Creek Software
have formed a “bucket brigade” to relay 5-gallon buckets of diesel fuel
up 17 flights of stairs at 75 Broad Street to refuel a generator
providing emergency power to the Peer 1 data center.

It is a decidedly low-tech, brute-force solution to the challenges at
75 Broad, the hobbled data center building in Lower Manhattan. Strong
arms and strong backs are replacing fuel pumps that normally bring
diesel from tanks in the basement to the upper floors of the building,
where the backup generators are providing emergency power to keep the
servers online in the Peer 1 data center. Those pumps shut down
yesterday when the storm surge from Hurricane Sandy flooded the basement
levels of 75 Broad, rendering the pumps unusable.

Peer 1 initially planned for a “controlled shutdown” at 10:45 a.m.
While the generators chugged on past that point, the company and its
customers developed an alternate plan -

“”We have 25 people stationed throughout the building to relay more
fuel to the roof once it arrives,” Squarespace reported on its status blog.
“The building now has powerful pumps clearing out the basement, which
we hope will expose the main pump lines — which would allow us fuel for
days. Hopefully our manual efforts, joined with the building, can see us
through. … ”Spirits are strong and everyone from Peer1, Fog Creek, and
Squarespace is working together.”

Squarespace acknowledges the effort is “not sustainable” over the
long haul. Both Peer 1 and Internap are working on long-term solutions
should the pumping and repairs require extended time. But the bucket
brigade reflects an unusual physical embodiment of the depth of
commitment to keep data centers online, come what may.

UPDATE: As of 1 p.m. Eastern the generator’s day
tankw as full, allowing the bucket brigade to take a 90-minute break.
But little progress was being made on long-term resolutions. “The
building’s first attempt at an alternative method for pumping fuel to
the 18th floor has failed, as the fuel pump wasn’t powerful enough,”
Squarespace reports.
“They believe they have sourced an alternate pump, but given the
situation in New York City right now, we’re in a wait-and-see posture.
Fuel- and water-pumps are in short supply.”

Perhaps more problematic is the stats report on the flooding. “The
basement is not draining at all either, despite the large pumps that
were brought in late last night. DEP and ConEd have been here for a few
hours. They fear a water main has ruptured somewhere and is pushing
water (and other stuff) into our basement as we pump it out. This is
pure speculation at this point.”