A data center story for the ages: the fuel bucket brigade

The effort to save a NYC data center following Hurricane Sandy is remembered in a new film.

The improbable story of the data center saved by a fuel bucket brigade is not told with fondness, even if there is a sense of pride and humor in its retelling.

Those involved recall the stink of diesel fuel in dark stairwells illuminated by flashlights. There was the sheer physical exhaustion of carrying fuel up 17 floors to deposit into a rooftop generator. There was the sleeping on floors and the lack of showers, and the unrelenting stress that their efforts could suddenly unravel and fail.

Then there was the broader weight on all involved about the impact that Hurricane Sandy was also having on their friends, family and on New York City itself.

The effort to save the data center during and after the October 2012 storm is captured in a documentary-style film that Peer 1, the data center owner, produced and showed Tuesday to a group of customers and employees.

The film, with photos of data center rescue efforts along with interviews of participants, doesn't attempt to romanticize or triumph the events. It just lays them out, and leaves the viewer with a sense of the sheer exhaustion watching the effort of the participants and the constant uncertainty of the outcome.

The idea of carrying fuel to the roof "seemed like a ridiculous idea," said Michael Pryor, president of Fog Creek Software, a large user of the Peer 1 data center, in an interview. "It just didn't see feasible."

Pryor's idea, retold during the film, was to bring some extra pumps from fish tank in his office to the nearby Peer 1 data center to see if they might help get fuel to rooftop generator. The proposal drew healthy laughter from the audience, something Pryor now chuckles at as well.

But his thoughts of using fish tank pumps helps explain how the data center workers and customers, in the early going, were grappling to come up with ideas for keeping a roof top generator that was burning 40 gallons of fuel an hour out of a 200 gallon capacity tank running.

Fog Creek makes a software management platform used by developers, and the loss of operations at the 5,000 square-foot data center at 75 Broad St. would have hurt its customers and its ability to generate revenue.