Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

Nerval's Lobster writes "Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel? In the wake of Hurricane Sandy, the challenges facing data center operators in the affected zones include pumping water from basements, waiting for utility power to be restored, and managing fuel-truck deliveries. And it's become increasingly clear which companies had the resources and foresight to plan for a disaster like Sandy, and which are simply reacting. Here's the latest on providers around the New York area." And remember, having fuel for machines sometimes only means it's time to start the manual labor.

You need a little more compression to light #2 fuel oil, the engine in the duece and a half [wikipedia.org], LDS-427 Turbo Multifuel I-6; LDS-465 Multifuel I-6; 127 to 170 hp (95 to 130 kW) will burn it OK, but they are small for data center usage. I'm sure there are marine engines off the shelf that are both large enough and able to run on both.

I don't understand why they're using deisel rather than natural gas. With natural gas they could have the generators on the roof. Illinois Secretary of State mainframes have two backup generators, both natural gas. I've lost electricity a whole lot of times, but I've never had a natural gas disruption in my six decades. Not even in 2006 when we were hit with two tornados and electricity was out for a week; I heated my apartment with the oven (the gas furnace needs electricity for blowers and theromostats).

Maybe "nearly every car" might be an over statement. But every car i have owned has the pump in the fuel tank. My 94 chevy pickup which i have replaced the pump is in the tank. my first car an oldsmobile had the pump in the tank. And my Taurus has the pump in the tank.

I was under the impression that a fuel supply was a standard part of the contingency plan for any data center.

I recently visited a new data center opening near me. The operator had contracts with several fuel suppliers that in the event of a power outage, the first one to get a full tank truck through their front gate got paid, and would keep getting paid for each additional truck that was needed. Any latecomers would be turned away, effectively making it an exclusive contract upon arrival.

You also have to hope that the trucks are allowed to travel. I worked at a DR company that sold contracts on datacenter in a trailer solutions. Many companies that got hit by Katrina declared, and not one single trailer was sent. The contract had provisions that fuel for the generators had to be available. There were going to be problems, so nothing was sent to assist them. Basically free money for the potential use of these trailers.

Driving tank trucks around any of the Boroughs seems like a pretty tough proposition even today. In a real disaster, plan to not be able to even call the fuel dealers and ask if they are coming, much less getting them through the main gate and down the service road.

Once refined, it degrades. Oxidation, bacteria. Algae. Amazing things grow in diesel. You can add preservatives, but these only go so far. Keep it too long, and it's unusable. If you don't use it, you have to dispose of it. There's a large disincentive to keep diesel around.

Once refined, it degrades. Oxidation, bacteria. Algae. Amazing things grow in diesel. You can add preservatives, but these only go so far

I wonder if it would be economical to manufacture a grade of diesel-compatible fuel that doesn't degrade, for purposes of long-term storage. It would consist purely of highly refined aliphatics, leaving no unsaturated bonds available for oxidative attack. Heavy metal content would need to be near-zero as well, as they could function as catalysts for breakdown, or nutrients for microbes. The storage air-space above it would be purged with inert gas or at least desiccated, again to avoid microbial growth.

If a liquid paraffin similar to medical grade mineral oil (but a lighter fraction) will keep a modern Diesel (yes, with a capital D) happy and survive long-term storage, then the problem is you: Go forth and sell.

What's the point in trying to keep these servers operating, and risking another fire hazard. Not to mention all the pollution and contamination. I'd be renting a bunch of cloud servers on Amazon, or move what you can to another location.

No. Anything entirely within a single datacenter or city does not fit any reasonable definition of "cloud." That's the whole point of it; no single point of failure, in case something like Sandy happens.

Have you noticed any outages on google, or youtube, or netflix, or any other geographically dispersed network? I haven't.

Everyone already knew that the on-site fuel supply is the limiting factor of power availability in a disaster. Even fuel delivery contracts mean nothing in a disaster or wide-spread outages - hospitals, EMS and other government services will trump the fuel delivery contract, if a hospital needs fuel, they are going to get the fuel that's been "guaranteed" for your datacenter.

There's no reason to spend big $$ creating a flood proof, earthquake proof, tornado proof, airplane crash proof datacenter in the middle of a city when you can have a disaster recovery site 1000 miles away that's not subject to the same type of disaster. (except maybe an asteroid strike, but there are few datacenters on the moon). No matter how disaster-proof you make your datacenter, mother nature (or man) will always find a way to create a disaster you didn't plan for -- even if that "disaster" is a typo in a router configuration file that takes down the network, or a contractor accidentally shorting out the emergency power cutoff switch wiring when bolting a rack to the wall.

If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.

If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.

Sure, there will always be datacenters in NYC, but that doesn't change the fact that instead of putting all your money into trying to build a datacenter impervious to all hazards, you're better off having a second site far from your primary site. Real-world constraints mean that you can't build that perfectly impervious datacenter when you're subject to real estate prices, building codes, fire codes, and Murphy's Law -- there will always be a disaster that the facility can't handle.

If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York.

I don't think OP was saying that you shouldn't have datacenters in NYC, persay, but rather that you'd be better off spending most of your "disaster-proofing" budget on off-site disaster recovery equipment/services.

I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.

So as you can see, this isn't a question of them not stocking enough fuel -- the cost of storing that fuel is prohibitively expensive.

And that, people, is why they didn't load up on fuel ahead of the storm. You can't simply pay $83 million a year for your data center to protect against a threat that might only materialize once a decade, and be severe enough to deny fuel deliveries or power restoration for that long of a time frame. Generator backup is a short term solution. There is no long term solution for disaster recovery, at least not one that's cost effective. Not in an urban setting.

The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services. Everyone else plans for a couple day supply and leaves it at that.

I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.

While I agree with your point in general, your math is off. I can see our own 1MW generator and 3-day fuel tank from my office, which is no where near the size of tank you calculated -- it should consume around 1/4 of a city block using your figures.

A week of fuel (24,000 gallons) is 3200 cubic feet [google.com], with a 10 foot high tank, that's only 320 square feet, at $100/month that's still a pricey $384,000/year. But since that 2MW of power is enough to power 1000 servers (assuming 1000 watts/server, half the powe

I just googled to source the numbers to get a ballpark figure. So yeah, undoubtedly they're off -- people spend months preparing reports for their data center on fuel consumption, storage costs, location, etc. I spent 20 minutes. But putting the numbers together shows that even if you lowball all the numbers by 50%, storage cost is still massively eclipsing fuel cost.

People on slashdot here were being 'armchair CTOs' and saying how if it was their data center, they'd have bought all that extra fuel. "Well

I just googled to source the numbers to get a ballpark figure. So yeah, undoubtedly they're off -- people spend months preparing reports for their data center on fuel consumption, storage costs, location, etc. I spent 20 minutes. But putting the numbers together shows that even if you lowball all the numbers by 50%, storage cost is still massively eclipsing fuel cost.

Sure, I understand the fuel usage numbers were an estimate, but your calculations weren't off by 50% - you were claiming that it would take an entire 264x264 foot office floor (70,000 square feet) to store a week's worth of fuel. I was pointing out that it only takes about 300 square feet - you were off by a factor of 233, not by 50%. Instead of an unaffordable $83M in rent and an entire office floor to store a week's worth of fuel, it only takes an 18 foot by 18 foot room and $400K/year. That's a huge di

"The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services."

Which is why I can't understand why some hospitals in New York City had to evacuate due to power loss. I heard that one hospital evacuated the goddamn NICU.

...

Whoever was responsible for disaster planning for these hospitals ought to be facing criminal charges.

Hospitals, even publicly funded ones, don't have unlimited funds to implement disaster recovery plans. If it comes down to reducing services during the 99.9% of the time when there is no disaster versus spending money to ride out the 0.1% of the time when there's a major disaster, spending the money to help more people now isn't necessarily worse than planning to evacuate during an uncommon disaster.

I'd question the need for a hospital to have 45 days of fuel on-site... keeping the lights on does little goo

Telling injured hurricane victims that show up in the E.D. "Sorry, dude, but this storm really screwed us up, man. We can't treat you." isn't acceptable.

Sure, I understand that a hospital needs to function for some time during/after a disaster, but 45 days of supplies seems like expensive overkill - do you have millions of gallons of water on site and your own treatment plant on-site since it's not likely that your municipal plant will withstand a 30+ day power outage.

Cell sites I worked in Africa run a pair of Cummins generators as their main power. In the unlikely event that both of these fail at the same time, there's a chance that the main grid might be working well enough to take over. But the fuel is the biggest priority on these sites.

At the moment, we have tankers on site at all of our NY DCs. 2 of them are on generator only, so we're topping up the tanks every 4 hours. The generators will need full overhauls when we're back on the grid properly, but for now we're keeping our clients serviced which is what matters.

I've heard this before, but I've never seen the reasoning - why do the generators need full overhauls after running for a few days? Aren't these the same engines that are in trucks/construction equipment and run for thousands of overalls without an overhaul?

I've heard this before, but I've never seen the reasoning - why do the generators need full overhauls after running for a few days? Aren't these the same engines that are in trucks/construction equipment and run for thousands of overalls without an overhaul?

When i worked at some remote construction sites, they had 1MW+ generators that went for over a year of nearly continuous use between overhauls. As I recall, they did oil changes and other routine maintenance monthly.

An emergency power generator might be run at a higher load than a continuous duty generator, but I can't imagine any large diesel generator needing an overhaul after a few hundred hours of operation.

If this was their first time, or it had been a few years, I would expect there to be mass hysteria and a general failure of DR plans. Mostly because DR plans are theoretical and costly to test and not tested very often. However I think Irene gleaned much useful information for those developing these DR plans which led to NY data centers being better prepared. I'd love to read the DR plans in 6 months and compare the new changes from lessons learned during and after this storm.

(Please keep in mind that I write this with a somewhat cruddy memory as to historical events.) I'm on the fence with how to feel on this one. Being SO dependent on computers and data centers is a relatively recent phenomenon. Not saying they haven't been important before, but it's really blossomed over the last 10 to 15 years. Prior to Katrina in the gulf region, I can't recall a storm of this magnitude in the last decade. (Although the same northeast region got hit with "Hazel" back in 1954) And this

And colo providers get driven down on cost as customers forget the previous catastrophic event. Every dollar spent enhancing a data center is passed along to the customer in some fashion. When your per-cab or per-foot pricing is higher than the competition--even if you can demonstrate more resiliency--it is a harder sell.

You can talk flood plains, gen sets, fuel locations and delivery schedules, CRAH and chiller redundancy, etc. As long as the SSAE16 or SOC 2 is current and there are no lingering memories o

This is very easy to plan for. Simply decide how far away your DR location needs to be. If the problem is fuel, then the DR plan was a shelter in place plan, and is going to suffer miserably for many reasons. Long term power outage caused by floods is very obvious, and will bring the exact issues that are being seen here every time. This is likely in a plan for all of these places. A fire would knock these completely offline, so there always has to be consideration of failing to an alternate location. The m

I was visiting 1 summer st once, and I asked the guy who set the place up "why don't you use natural gas? You can just get it from your local utilities, or have it delivered by truck." and he looked at me with the "you poor idiot" look people give me some times. Then he said "Nobody uses natural gas because you can't tell if the supply upstream has been disrupted."

I didn't feel like explaining "no, dude, you pump it into a pressurized tank and monitor the flow, *and* you have a diesel generator as a backu

Having large quantities of pressurized natural gas can be an explosion risk. Diesel's just a fire risk. Storing pressurized natural gas likely involves a lot of regulatory issues that diesel doesn't.

Also, natural gas is not very efficient volumetrically: diesel has 37.3 MJ/liter. Natural gas at normal pressure has 0.0364 MJ/liter. Even compressed to 250 bar (3,600 psi), a non-trivial pressure, it only has an energy density of 9 MJ/liter. Liquified natural gas at -160C has an energy density of 22.2 MJ/liter.

Lastly -- why the heck did these guys put the pumps *outside* the tank? Why not put them in the tank?

ever tried to service a pump that's inside a tank? real pig to replace them... far easier if they're outside the tank as you can have shut-off valves either side and isolate them while replacing them... you don't have to drain the tank and purge it of fumes first before accessing the pump...

SOP on gasolineautomotive fuel pump replacements is, if possible, to do so with a full tank to minimize the risk of badness (gas itself doesn't burn, while gasoline vapors both burn and can be explosive).

What is it about a large volume of Diesel fuel that makes this different in any meaningful way?

Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel?

I assume everybody knew. I knew and I am not even in the data center business.This is not just some afterthought that I suddenly thought: well that seems logical.I talked to people who worked in data centers and when we talked about contingency, they all had examples where there was an issue with the fuel supply. Most where the same story, probably as it never happened to them, yet

One of my buds is IT director for a company that resupplies generators. The logisitics for it are crazy as you route trucks on available streets, deal with priority of the customers (hospitals front of the line) and optimize resupply into mostly empty tanks before they actually empty, etc... And you have >10 days of this 24/7 after a storm.

After Hurricane Allison here, some companies in this sector went out of business before the power came back on. We were better prepared for Ike, but I think that's because facilities are more willing to sign contracts with the pricey but reputable businesses.

This is the most expensive way to get fuel: you need a massive amount for a short time, and you need it consistently during that time.

This is the most expensive way to get fuel: you need a massive amount for a short time, and you need it consistently during that time.

The problem has been getting it from where you fuel up the tanker, to where it's needed. The fuel's cheap comparatively, but the trucks aren't really off-road vehicles. When you've got flooded streets, streets lined with debris, or streets that are completely gone, the trucks really can't make it in to resupply.

I think you mean Tropical Storm Allison. Man that thing flooded the hell out of Houston. Then went on to travel over land all the way to the mid-Atlantic States flooding pretty much everywhere it went. Following this the name Allison was retired from the Hurricane name list. It remains the only name to be retired from the list for a named storm that never reached Hurricane strength.

30 years ago, I retired from the Navy, along with a friend who was a Machinist Mate on Navy submarines. I didn't see him for a couple years. Turns out that he had spent 10 of those years servicing batteries on diesel submarines, and ended up servicing large batteries for the phone company -- about the same size batteries that had been on the boats. He switched to servicing the diesels (because of his prior experience on submarines with their diesel generators), and one of the first to deploy natural gas fired diesels for sustained power, first for the phone company, and then for a local utility.

There's the solution. Don't pipe flammable liquid to the top floor, pipe the natural gas. Contrary to popular belief, it is a safer fuel, and requires less maintenance. Automatic shutoff valves work better, less explosive volume released on a tubing breakage, and it doesn't rot the pipes like liquid diesel fuel, disperses to atmosphere in the case of a leak, and doesn't make everything around it flammable, when it does leak.

Engines run at least 4 times as long between between overhaul cycles, and it doesn't dilute the lube oil.

Natural gas is going to be a HUGE change for this countries infrastructure, both in common usage, as well as emergency failover services.

This guy has made tons of money, by the way. Has a condo and car in San Diego, LA, Seattle, New York, and Atlanta -- cheaper than hotels and taxis, and doesn't know what else to do with his money.

Natural gas is going to be a HUGE change for this countries infrastructure, both in common usage, as well as emergency failover services.

I'm generally enthusiastic about natural gas, primarily because they can safely run unmanned for extended periods, but let's be honest, there are plenty of problems... Not only do the generators need to be bigger and more expensive, but you've gotta have a back-up plan in case those natural gas lines are either damaged or shut-down.

At my work we had our generators on the roof above 6th floor. It had a small tank that can run for a couple of hours. In the basement, a larger tank that had fuel for 2 days.Sadly when the power went away, so did the power to the pump that were supposed to lift it to the root. So the our electrician all of the suddon got busy.

At another workplace we had two huge generators with flywheels for starting them, plenty of fuel and even our own redundant line to the powerplant some miles away. Sadly when the insta

This is one of those situations where the invisible hand of the free market fails miserably:

Company A: plans properly for contingencies, but has to charge $10 more per customer.

Company B: thinks they have planned for contingencies as well as A (but accredit the cheaper pricetag to their managerial prowess), makes the same claims about uptime as company A, and undercuts company A.

Company A goes out of business as B steals all of A's customers.

Disaster hits, but due to company A having gone out of business 2 years ago, B can legitimately say "no one could have planned for this!" and likely gets away with it...

Which makes me question the wisdom of urban datacenters in the first place.

It's nothing particularly special to the datacenter being in a city. Get a big enough disaster and you will be knocked off the net, wherever you are and whatever steps you take to prevent it. The best mitigation strategy is to have geographically distributed datacenters that can run replicated services that are structured so that losing one site entirely is not crippling (though it might hurt a lot). This isn't a simple thing to do — for most types of service, you have to design the overall service from the ground up to work that way — but it's not too bad. (And at least with the Cloud you don't have to build your own collection of geographically-separated datacenters or pound your head against the problems you usually get with hosting providers when you're desperate for service in a hurry.)

Mind you, rents in NYC are so high that I'd guess that nobody's going there for the cheapness of the service. Instead, it's probably for the proximity to the Wall St. exchanges. The fun thing with that? Any datacenter which could satisfy the latency requirements for people in that business would have been deeply hit by this particular disaster, and all the places that could easily offer continuous service this time would never be able to normally compete for this particular type of business. To double the fun, the exchanges were closed for two days anyway. That's modern capitalism at work...

Think. It is a fire hazard. As in "it starts leaking because of event X happening". And where does stuff go when it leaks? Yes, it goes down.

That is why fuel is stored DOWN in the basement. *Water*, on the other hand, you can store on the roof.

If you don't want to worry about your tank getting flooded, then make it water tight with air inlet well up high out of the water. There are safe and reliable ways of doing this without storing your fuel on the roof!!

You also need to make sure to bolt the damn tank down. Fuel and air are more buoyant than water or mud. More than a few flooded gas stations have suffered damage only because the tanked popped out of the ground.

I agree. I recently has to do a disaster recovery plan for a law firm I administrate. They wanted to be able to weather any event without disruption. I took disruptions that I knew of within the last ten year, and expanding to situations that has happened in other parts of the town. Flooding wasn't really an issue because they are on a hill that would require most of the state to be under water before they were. The biggest concerns was power and internet outages, something happening to the building making

I am busy making sure that these servers that transmit data to hundreds of millions of people keep going no matter what kind of pounding it takes or how drenched in fluid it becomes, just like the chicks in the videos that the servers are transmitting.

First, building an electricity-hungry data center in NYC, with its notorious electricity problems and in the path of coastal storms... well... let's just say it's not the brightest star in the cosmos of ideas.

In fact, it kind of reminds me of... what was it? Oh, yeah. A major Southern coastal city that was built... mostly below sea level! Yeah! That's the ticket!

I spent some years over there, mostly in the Philadelphia area, and that entire section of East Coast had pretty regular power outages (sections of it, that is... not the whole thing at any one time, of course).

Compare that to here in the (relatively speaking) West, where electrical outages in any one place happen maybe about once every 5 years, and outages that last more than an hour or so might happen roughly once every 20 years or so.