Posted
by
timothy
on Monday September 17, 2012 @04:16PM
from the more-and-bigger-flywheels dept.

1sockchuck writes "Data centers operators often tout their diesel backup generators as a symbol of their reliability. So why does Microsoft want to get rid of them? Microsoft says diesel generators are 'inefficient and costly' and is looking at alternatives to supply emergency backup power for its server farms, including fuel cells powered by natural gas. One possible option is the 'Bloom box,' which both Appleand eBay are using in their data centers (albeit with biogas as the primary fuel). Bloom is positioning its fuel cells as a way to forego expensive UPS units and generators, using the Bloom box for primary power and the utility grid for backup. It's a pitch that benefits from the current low price of natural gas." (Microsoft would like to stop using so much water, too.)

Fuel cells may be more efficient, but the diesel generators are already in place.

Existing Diesel engines can be set up dual-fuel too, where they may start using the diesel, but can switch to natural gas. That allows either municipal natural gas service lines to supply power, or tank, which can be fixed in place or installed on a truck to supply fuel. Some generators are probably built to start up and run on either fuel as well.

That would allow at least two kinds of fuel to power the generator, with multiple delivery methods.

If you want to reduce pollution, focus on your primary energy source, not the backup. A data center probably uses 100 times more energy from the primary source than from the backup.

Difference is, from an environmental permitting perspective, the data center isn't responsible for the power plant, but IS responsible for their own generators. And permitting even small units are becoming somewhat of a hassle in the states. When they do run, they can really belch out nitrogen oxides in surprising quantities; despite the "emergency" nature, some environmental regulatory agencies will look at these engines as "operating all the time" units, and will regulate them accordingly strictly.

That makes it bad for the American economy and interests. It will sacrifice jobs in the energy industry in the name of some liberal, unproven, uncientific, utopian, dreamy, touchy-feely ideals and global warming paranoia, proposing that investing less in American energy is necessary and even productive. People who buy diesel, gas, and oil products are Americans who believe in invesing in power for their productivity, enterprise, and lifestyle. It is simple math, the power-to-weight ratio of any combustio

T-Mobile Sidekick Disaster: Danger's Servers Crashed, And They Don't Have A Backup

Wow. T-Mobile and Danger, the Microsoft-owned subsidiary that makes the Sidekick, has just announced that they’ve likely lost all user data that was being stored on Microsoft’s servers due to a server failure. That means that any contacts, photos, calendars, or to-do lists that haven’t been locally backed up are gone. Apparently if you don’t turn off your Sid

If they replace backup generators with some alternate technology. I hope that they actually make sure it is reliable first. And that it stays reliable over time. (eg, three years later, you suddenly need it, does it still work?)

If the alternate energy source is renewable and reliable it should probably be the main source of energy rather than just for back-up purposes.

If on-site power generation becomes commonplace and more large customers do it, is there a possibility that the power company will oversell their capacity enough that a large natural gas outage will also take down the power grid?

If there are are 5 datacenters in a particular area and each has the capacity to draw 20MW of power, but usually generate their power via natural gas, can each facility be assured that the power grid in that area can supply 100MW of power to keep all 5 datacenters running when a maj

Most data centres normally get their primary power from the main electrical grid. Backup is normally via batteries for almost instantaneous but short term (15 minutes to an hour) electrical supply while longer term (greater than 24 hours) electrical supply is normally done via diesel generators. Of course you do have some sites that can generate power via some sort of solar energy (eg. wind, photovoltaic,..) or even natural gas. The problem you have from alternative resources is they have some drawbacks su

If they replace backup generators with some alternate technology. I hope that they actually make sure it is reliable first. And that it stays reliable over time. (eg, three years later, you suddenly need it, does it still work?)

They're not just replacing it though; they're flipping it. The primary power source will be on-site generation, with the backup being the grid. So reliability becomes less of a concern -- the onsite system is running constantly, meaning any faults will be found pretty quickly and it'll drop to grid power while you troubleshoot those. Meanwhile, the only maintenance they need to worry about on the backup system is a couple wires and the actual switching system. It's not like a diesel backup system where you have to run it every couple months anyway.

This seems like it would be a lot more reliable than the usual system, even if their new natural gas generators aren't. Because you'll never have a situation where you go to switch to backup only to discover some part on the generator broke while it was sitting idle.

Of course, they're going to end up spending more money on manpower to keep their local generation systems maintained, since they'll be running all the time -- which makes me wonder if that's what would really be driving any improved reliability.

Around here, there are HUGE liquid storage natural gas tanks that hold weeks to months of fuel. The distribution system just works based on the naturally high pressure of the gas, no pumping involved. Natural gas still works, even with weeks of power outages.

Around here, there are HUGE liquid storage natural gas tanks that hold weeks to months of fuel. The distribution system just works based on the naturally high pressure of the gas, no pumping involved. Natural gas still works, even with weeks of power outages.

Yes and no. Natural gas lines rely on compressor stations to maintain the pressure. Some of these actually run on natural gas themselves but others are powered by alternate fuels. Also, depending on the condition of the line and weather during a power outage, a freeze-off due to accumulated water is a real possibility. But, your core point is correct - natural gas service is mostly dependable regardless of grid power and a good backup option.

I grew up in a rural-ish bit of England, and there were occasional power cuts -- perhaps every 6 months or so. The natural gas never failed -- the electronic ignition for the gas cooker wouldn't work, but it could easily be lit with a match. A bland roast dinner can still be made in the dark.

Since the gas is pressurized, the pipeline itself stabilizes for interruptions. Pressure drops slowly, but as long as you have a compressor at your end you can ride through.

The big impact to reliability is actually the requirement for automatic shut-off valves after the PG&E pipeline explosion in San Bruno. The pipelines are reliable today because they are almost impossible to isolate. Once that goes away you will either need a lot of local storage or a second source of fuel. The grid won't be any m

Who only tests their generator "every couple months"? Ours exercise weekly and have quarterly PM's from Caterpillar (none of our datacenter generators are Cat's but they were already doing the work for the building generators and it's not like their diesel mechanics know any less than anyone elses).

Please tell me those are just for the datacenter and not the ones supplying power to life critical systems (please!). I mean if by test you mean run full load off them then I understand, but if you mean your generators are never exercised then it's no wonder at least one of them fails during your annual run.

To be fair, they've improved data storage reliability as a result of incidents like these. Now, when you provision a storage account under Azure, there's a check box (checked by default) that turns on "global backup". When on, any data written to storage is fanned out across two geographically diverse data centers.

Say what you will about their technical mistakes, Microsoft does learn from them. As far as their decisions when it comes to product development and marketing, though, they don't appear to have

Of course it's reliable - it's time tested technology as used on Apollo 13:)On a more serious note, generators require a lot of care and feeding while an elaborate battery requires less once you consider a long timeframe. If you don't have an experienced diesel plant mechanic onsite there's not much point having diesel generators

If you don't have an experienced diesel plant mechanic onsite there's not much point having diesel generators

Why would you need an experienced diesel plant mechanic on-site? That is like saying you need a experienced motor mechanic sitting in every car that is on the road. Yes if a generator fails you would have to get a mechanic out to the site to do minor repairs but you don't have to have a diesel plant mechanic always on-site. Normally a data centre has two or more generators and if a generator goes out it is possible to get support generators delivered on-site within a few hours and integrating them into the

That said, in all seriousness. If they replace backup generators with some alternate technology. I hope that they actually make sure it is reliable first. And that it stays reliable over time. (eg, three years later, you suddenly need it, does it still work?)

You are correct. It really does not matter what method you use to provide a backup power source for your data centre providing it is reliable. The problem all data centres have is normally high end management in that it is very difficult to get them to agree to regular testing of the power backup system since there is always a small possibility of a failure. It is essential that regular testing is carried out (say 3 to 6 months) otherwise when the main power grid goes down you could have a catastrophe on yo

You can buy diesel in the UK for stationary engine use without paying tax ("red" diesel) or if you buy enough of it to make it worth jumping through the hoops form-filling you can get the Vehicle Fuel Duty and VAT paid at the pumps refunded as it's not being used in a road vehicle.

Friends who own a coastal barge/houseboat buy red diesel from the local marina; I bought them a hundred gallons for their honeymoon cruise as a wedding present.

With the north sea fields, you used to be an exporter, but they peaked and declined a couple years ago so you're an importer. Its only going to get more costly in the future, plus or minus economic decline.

Only so long as the Greenies In Government prevent them from using the massive amounts of shale gas that's just waiting to be tapped.

There is nothing wrong with diesel emergency backup generators (keyword EMERGENCY BACKUP). They are easy to run and maintain, assuming you have a basic understanding of engines. Check the oil and fire them up every 6 months or so, plus a mechanic taking a good look once a year or so. They are large, so it easy for someone to get in there and replace parts. You can fuel them by getting diesel from any gas station in the area. And if they break, there are thousands of mechanics which can fix them, local to pretty much anywhere that has civilization.

The "Bloom Box", on the other hand, is a specialized piece of equipment which only a few people know how it actually works. It uses a niche fuel source, not going to find that at a Shell station down the road.

I sure hope you fire them more often than every 6 months! Weekly maintenance runs make a lot more sense. Leave an engine unused for 6 months over a long period of time and you will create more maintenance problems than you can imagine.

You're right, probably depends a lot on the climate as well. Weekly is best, assuming that someone in your data center knows a little bit about engines. Don't see any reason why a generator couldn't turn itself on every week and e-mail necessary information like oil pressure/content, running temperature after XXX minutes, fuel level, etc. If it hasn't been made, I will be in my garage for the next month, hoping it isn't already patented.

On a related note, I just visited an AT&T international data center a few weeks ago and their backup systems are pretty phenomenal. They opt for once a month, instead of once a week, to exercise their generators. And when they do, it is a full test with no grid power. However, they rarely have to schedule a monthly test because they usually end up dropping off the grid at least once a month at the request or

Having worked at a small high-availability data center I can assure you that AT&T does it right. Just starting the generators is not enough. You must also test your UPS systems, line sensing equipment and the long and complex line of electrical equipment that must work if you are to get a clean handover from the power grid over UPS to generator. Finding a failure in the chain is not nearly as painful when you can fall back to grid power while troubleshooting.One of the more sneaky things to do to a faci

The need to test that stuff isn't so big when you use it every day. AT&T has rooms with a thousand or so batteries in each. All power, whether grid or generator passes through line sensing and smoothing equipment and the batteries before it hits the raised floor.

Yep, the weekly maintenance cycle is how we found out the idiots hadn't specified oil or battery heaters for our backup generators, MUCH better to find out due to a weekly test than to find out when a blizzard takes out power.

I think our protocol is monthly, though it could be quarterly, I'm not in charge of the generator. Diesel fuel is very shelf-stable, and since there's no ignition system to futz with it's just a matter of ensuring that the starter cranks and the injector pump isn't leaking.

This isn't like a car. It doesn't move around so it doesn't suffer the abuse that the road and travelling gives, nor does it have a transmission, so there's no gearbox to wear, and since it's designed to spin th

Yep, weekly is a great schedule, and as I posted upthread we only found out that our generators weren't properly winter specced due to a weekly test, a monthly test might have missed it if it had happened on a couple of warm days during the year.

Then hope that you live close to one of the 519 CNG stations [energy.gov] in the country. Compare that to the amount of diesel stations in the country, BP alone has 367 diesel gas stations [mybpstation.com] in the Los Angeles area alone.

I hope you are joking. Did you think a datacenter technician would take a gas cylinder, go to a CNG station, refill it and bring it back to the data center? Datacenters use enough natural gas to have a purpose build huge storage facility and sign direct contracts with the gas companies to maintain and refill the gas storage,

Yep, in fact we're producing so much of it that the biggest barrier to lower prices right now is pipeline capacity. Heck right now Mexico is shipping crude to the US, using cheap natural gas to crack it, and reimporting the processed fuel because it's cheaper than using some fraction of the crude to fuel the cracking process.

For example, GP mentioned being the Los Angeles area. So Cal is very earthquake prone and, it turns out, pipelines don't react well to having the earth they're laid in shifted three or four feet in one direction so onsite facilities make a bit more sense. I live in the upper midwest in a mitten-shaped state where the trees are just the right height and the earth doesn't tend to quaketh as often, so the places I've worked were all natural gas pipeline fed.

We do have natural gas by pipe in the US, but coverage is spotty. As a matter of fact, where I am from in California, there are no natural gas pipes. Where I live in Maryland, one block can have natural gas and the other doesn't. 5 miles can make all the difference in which utilities are available.

Natural gas is most prevalent in areas which have colder winters, because it is such an effective heating source. In some areas (Monterey Bay California area), the winter never gets cold enough to justify inst

What's the point of an emergency backup generator that is dependent on a utility company? The whole point of a backup is to be able to run your own facility without their help. What if electricity and gas lines are down? Whatever a backup power supply, it should be 100% independent of any public infrastructure.

Actually I hope that more datacenters start using natural gas to power their backups. That means that the datacenter I work for will be able to run for months without being dependent on a public u

Actually I hope that more datacenters start using natural gas to power their backups. That means that the datacenter I work for will be able to run for months without being dependent on a public utility company

The actual costs of generating your own power rarely match the cost of receiving power from a reliable customer power grid. Of course if the customer power grid is unreliable then the data centre may have to generate it's own power. which is not going to be cheap. One point I should make about natural gas is that it is a non renewable energy resource and many places don't have the infrastructure to pipe the gas to a required location.

You are not dependent on the utility company, but on the gas company (Shell, etc). As far I understand, diesel is also supplied by trucks by the same gas company. I wonder if the regular CNG can be used directly on these fuel cells though, impurities can pretty much kill the cells.

Let me rephrase. What's the point of an emergency backup generator which has a fuel source that is dependent on a single public utility company? I'd rather have the ability to call around during some sort of emergency event and find someone to bring out a tanker truck full of diesel than wait around for a utility company to get around to fixing their infrastructure.

Sorry but Microsoft isn't running down to the corner store for fuel, heck neither is my little S&P 500 company. We have trucks deliver fuel a tanker full at a time. With all my generators going full bore I'm burning over 100 gallons an hour, hardly something you can deliver via gas can =)

You do have a point, although I can foresee problems obtaining diesel fuel too. In any case, I'm not arguing the guy's general argument in favor of diesel as a tried-and-true tech. I was merely pointing out his factual error in referring to fuel cells as needing a "niche fuel source". He was clearly assuming that all fuel cells are fueled by hydrogen gas, which is not the case.

There are problems with your post, but I'm not going to bother with them — I get very tired of responding to posts that disagree with something I didn't actually say. If you read my post again (without reading into it) I was simply pointing out a factual error.

I'm guessing you aren't aware of what a Tier-2 engine is. For that matter... what a Tier-4 engine has. The complexity of the engines today is a lot more than the old tractor engine connected to a synchronous generator of the olden days.

The real prob, assuming you live outside 3rd world areas, is the local electric power co is more reliable than transfer switches and generators.

Its legendary in the telco biz how many outages faulty transfer switches and generator testing "accidents" cause.

Local power causes many fewer outages, but the PR of "we're down because of no generator" "competitors have gens" means we have to lower our quality of service by installing generators, which is too bad. The customers are so dumb they'd rather have 10 hours of outage per decade due to x-fer switch issues than 1 hour of outage per decade due to power failures.

The customers are so dumb they'd rather have 10 hours of outage per decade due to x-fer switch issues than 1 hour of outage per decade due to power failures.

Guess I must live in the third world, because we've had far more than 10 hours of power outages in the last decade. We had about eight hours a couple of months ago due to tree branches taking out power lines in a thunderstorm.

Around here all the power lines are underground. That is one reason to pick a civilized place to put a data center.

Around here all the power lines are on poles except for shopping centers and new residential subdivisions. Regardless, our data center was recently out of power for several days due to a large storm that knocked out power to a few million customers on the east cost of the US. Our natural gas generator kept things running like a champ.

Yes seen that and the diesel generators worked flawlessly however the power switching equipment failed and 30 minutes later we had a silent data centre. Of course we were thankful that we did not have a gas pipeline cut.

Depends on where your data center is located. I was in Cincinnati over the summer for job training, and our hotel had over 20 total hours of blackout over three instances -- and I was only there for six weeks! Even the hotel's backup power didn't last through the one. Not even the emergency lights were lit by the time we finished trying to drink our way through the outage in the parking lot and decided to head in:)

Of course, that was the worst of the three, and we were fairly lucky there -- there were part

We had more than 10 hours of power outages from failure in our datacentre just last year, not decade. and we are in a major city. mains power is vulnerable to stuffups, backhoes, flooding and all manner of incidents that DO occur even in a perfectly maintained system. Incidentally of those 10 hours of power failure at least 4 hours of it were complete outages as the backup generators failed due to fault transfer switches once and then a faulty building UPS, but hey that is still way better than 10 hours.

One advantage is you can schedule the testing so that the accidents occur off peak hours. Another is that the outages are then under your control - which is mostly a psychological preference, but the fact is the customer is much happier hearing "Sorry for the inconvenience, we're working on it" than "It's the power company's fault, we're not sure when they're going to have it fixed, and we can't do anything until they do".

In Europe (and most of Asia) we are blackmailed by the Russians over the price of natural gas so Diesel fuel is a much safer bet.
Methane has also the disadvantage of being difficult to store as you can't liquefy it at ordinary temperatures like propane.

Buzzwords aside, a distributed server farm ("cloud") has redundant power supplies (in addition to redundant everything else) by simple virtue of geographic distance and replication. Maybe it is not cost-effective to have "uninterruptible" power at any single location. Just let this site go offline for a while as the others pick up the slack. You already need this redundancy to handle other failure modes anyways.

FromTFA:"...2.46 million gallons of wastewater produced annually by the typical data center, which must be pumped offsite and treated..."

Why does cooling water used in a typical data center need to be treated?Isn't it pretty much just flowing through stacks and carrying heat out to radiators or cooling towers? What in the process of gaining/releasing heat requires subsequent treatment?

Or is it just that the cooling systems haven't even been designed with the care needed to ensure the water flowing through

Cooling water isn't 100% water. It has contaminants. It may have coolant depending on the plant. The pipes that are fed aren't free of contaminants either. Dirt, rust, debris, etc. Now a plant could use ultra-pure water but it's not worth the cost of purifying if it's used for cooling. After a while the level of contaminants may build up so that the water is less effective and there may be danger of clogs. Also the amount of coolant may need to be maintained. Thus the water is dumped/replenished.

If Microsoft wants to do something useful in this area, they should build an OS that can respond to a "power is failing" signal and reliably get to a safe state in a few hundred milliseconds. Then put enough capacitance in the power supply to provide a second or two of shutdown warning. With that, transient power failures are no problem, nor are short delays during transfer from line power to backup power.

good luck using software to write data onto disks that have taken themselves offline

That is why data in Enterprise Storage Arrays is cached before sending to physical disk. Cache storage on Storage Arrays can be in the order of many GB as well as having redundant internal battery supplies so when the disks are finally brought back up data resident in the cache is then written out to disk.

Is Natural Gas more reliable than electricity? I'm assuming that they don't have huge CNG tanks to keep their generators running if there's a natural gas interruption.

I can believe that under normal circumstances the natural gas is an excellent backup to electricity since there are many failure modes that can affect one service but not the other, but in a disaster (hurricane, earthquake, flood, etc) or widespread power blackout, is natural gas going to be more reliable than electricity?

If you are going to use Natural Gas which is a non renewable resource to generate electricity then you need a Natural Gas power plant and when you are looking at MW outputs this is not that cheap. Cut a gas line and I think I would like to move well away from the area, since one spark and boom. Conversely cut a power line and you don't want to touch it until the line has been isolated which normally takes a few seconds.

Bio-gas requires a source (a trash dump effectively) and a lot of processing (drying, etc.) to be burnable in a Bloom Box. There's little evidence companies have gone through this trouble, it's more likely they are just pumping natural gas in.

Perhaps if those boxes keep running without needing any maintenance or fueland nobody invents anything newer or more efficient in the meanwhile,in 90 months they'll save $1M, and in 360 months they'll make up theiroriginal investment.

In all seriousness I'm glad the're doing it. Progress can't happen until someone puts forth the capital and risk. Surely the parties involved can afford it even if it utterly fails and 100% of the investment is lost. At least someone is doing something tangible that could lead to other, perhaps better developments in the future.

The reason datacenters use diesel generators is fuel availability. In an extended outage, you can drive your pickup to the nearest station in operation, and fill up all the 55-gallon barrels you've got. Obviously calling for a fuel truck delivery is better, but diesel still gracefully degrades better than anything else.

The cheap-ass datacenters already use natural gas generators. They're cheap, clean, and require much less maintenance or other labor. But any idiot with a backhoe can knock out the gas li

It seems logical to me, to have a natural gas backup turbine do double duty during high electricity consumption times.

I don't know, a data center's power consumption will fluctuate some with lulls and peaks in traffic, but since the bulk of business usage is during the late morning and early afternoon, and the bulk of residential usage is from mid-afternoon to evening, I'd think that using something like solar to supplement the grid would make more sense. That wouldn't put as much wear and tear on your emergency generators as they'd be running much less often, and if proper maintenance protocols are followed then the gene

They do. Some data center operators get a deal on their electricity if they run their backup generators during peaks, reducing the load on the grid. As a bonus, it's a cheap readiness test of their backup power systems.

If one has a true distributed/cloud computing infrastructure, if you can survive from a datacenter or two going down why even have backup power?

I am wondering the same thing. It seems to me that this is one of those cases where legacy thinking hasn't evolved. You probably still want some UPS so that you can shutdown smoothly, but negating any sort of long term outage shouldnt be part of the modern "cloud" plan.