Our data center/server room has two different utility sources, same utility company, but our building is big enough that we have power coming into the building from both the north & south ends and the two circuits are independent of each other. The south end feeds most of the building, except the far north end and is covered by a whole building generator. The north end source, has a small 30Kva generator that feeds the IT offices and a second circuit for the server room. Behind each server rack we have two 20A circuits, left is whole building circuit, right is IT circuit. In the bottom of each rack, we have a single UPS 2000va that connects to the IT circuit PDU. Their only purpose is to clean up utility power and keep things going for the few seconds it takes for generators to kick in. All critical servers have dual power supplies and connect to both sources. In the Comm rack we do have some single power source devices, but have a power transfer unit that connects to both circuits with a 1000va UPS on one circuit. We don't have any outages during Sunday generator tests.

Possibly overkill, but before we built the server room we had to make a lot of trips in due to power issues.

We have a 40kVA 208V 3-phase line interactive UPS with a large bank of batteries that protects our server room. Under the current load, it's enough to keep us up for about 12 hours. We also have a generator that kicks in automatically, so the UPS really just an expensive power conditioner that keeps the server room power on long enough for the generator to come online. The generator is fueled by natural gas, so unless there's a pretty major catastrophe, we're covered.

Our building has 1 power feed. We have 2 generators that power the entire building. We can run using a single generator if necessary. We may need to shed some of the HVAC load if necessary.

Behind the generators we have two large Liebert UPS units. Each UPS powers 1/2 of the UPS circuits for the building. Every workspace has a circuit on on of the UPS circuits.

For our call center and IT department, we have circuits from both UPS available. Workstations are distributed between the two UPS units, and in event of a UPS failure only half of the stations will go down initially, and they can be moved over to the remaining UPS. We have considered putting call center stations on automatic transfer switches.

The data center case a power bus system from Starline. We provide 208 and 110 voltage above the racks, with a separate bus providing power from each UPS.

Each UPS source in the data center is color coded. One is black, the other is red. Each each has 30A 208V PDU; one for black the other for red.

Only the correct colored cable is allowed to be connected from PDU to the equipment power supply. This makes it really easy to see that everything is correctly attached to A and B power. Each PDU has 1 color cable attached. Each piece of equipment has 2 color cords. Done.

For equipment with a single power input, it is attached to an ATS with 110V power. The ATS is of course attached to A and B power on the Starline, using red and black cords.

I ordered red PDU c13/c15 cables from stayonline.com. For other cables that are not available in a color, such a the built in cables on PDU, I wrap them in some red electrical tape to designate as a red cable. Super cheap but effective.

OP has a pretty nice setup, but if a UPS fails in any of the racks, then that equipment is vulnerable to power loss during a utility outage. IMHO it would be better to have both power feeds UPS protected. Also be sure that the UPS units are getting proactive maintenance.

@Sean, in our previous data center we had the entire data center on a single UPS. Thatbwasnuntil we had about 3 separate instances where UPS took down the data center. We had UPS fail due to batteries, and at least 2 instances where work in the building caused a short in a circuit leading to the UPS to trip its breakers. I would consider that single UPS a big SPOF. If you can't have 2 UPS, IMHO it would be better to use utility power as a second power input for your equipment.

It sounds like you have redundant power paths to your equipment. The only thing I might suggest is a UPS on each power source and the equipment plugged into an Automatic Transfer Switch fed by the dual UPS units. The Automatic Transfer Switches would be helpful with your single corded devices, but it sounds like you already have one in that setup. What model UPS units are you currently using? If they are APC units, I would be happy to recommend Automatic Transfer Switches for the UPS units if that is something you would like to consider. For warranty and any equipment guarantees, it is best for you to stick with one manufacturer for the devices in your setups.

I’d check with the building engineer to see if an elevator is on the whole building generator circuit. Also worthwhile to schedule a building power outage test so you can verify that the generator can carry the load.

Depending on how large of a building you are in, there may be (power hungry) surprises on other floors.

Our data center/server room has two different utility sources, same utility company, but our building is big enough that we have power coming into the building from both the north & south ends and the two circuits are independent of each other. The south end feeds most of the building, except the far north end and is covered by a whole building generator. The north end source, has a small 30Kva generator that feeds the IT offices and a second circuit for the server room. Behind each server rack we have two 20A circuits, left is whole building circuit, right is IT circuit. In the bottom of each rack, we have a single UPS 2000va that connects to the IT circuit PDU. Their only purpose is to clean up utility power and keep things going for the few seconds it takes for generators to kick in. All critical servers have dual power supplies and connect to both sources. In the Comm rack we do have some single power source devices, but have a power transfer unit that connects to both circuits with a 1000va UPS on one circuit. We don't have any outages during Sunday generator tests.

Possibly overkill, but before we built the server room we had to make a lot of trips in due to power issues.

This sounds like a fairly robust setup, well done. You've gotten some good feedback already too. Here are my thoughts:

You can always layer in more fault tolerance, but there comes a point of diminishing returns. For example, you could add an ATS and another 2000VA UPS to each rack for redundancy there, but is it worth it? What is worth it will depend on your goals. If you want to become a Tier-4 data center, all the critical infrastructure should be 2N+1 and that can get expensive while only marginally reducing downtime risk. Worded differently: it can cost a lot of money to go from 99.98% uptime to 99.99% uptime, if it's worth it is a business decision. We recently published a two part blog on redundancy in the data center and beyond if you're interested in more detail.

Also, from an operational resilience standpoint, if you're covered from a power perspective (and it sounds like you are), the additional investment may have better utility if applied to something like cooling/environmental management and/or out-of-band management (e.g. getting into the network with a console server like the B098-016-V using 4G LTE in case of a network outage).

I'd also agree with mountainbiker​'s suggestion to test the setup when power is lost to the entire building if you haven't already.

+Mountainbike Thanks or the comments. No elevator. We haven't scheduled any tests, but have multi-day outages that give it plenty of testing. The building overall is a huge warehouse. Around 200,000 SF of warehouse and another 50 - 60,000 SF of office space.The only drain on the building generator is AC and reel spooling machines that can handle 20 - 30,000 pounds.

David (Tripp Lite) The main AC unit for the server room are on the building circuit. The smaller backup AC unit is on the IT circuit. We have tested the backup unit and it won't keep the room at 66 which is where we keep the main unit set, but we haven't registered above 68 even on the smaller backup unit.

+Mountainbike Thanks or the comments. No elevator. We haven't scheduled any tests, but have multi-day outages that give it plenty of testing. The building overall is a huge warehouse. Around 200,000 SF of warehouse and another 50 - 60,000 SF of office space.The only drain on the building generator is AC and reel spooling machines that can handle 20 - 30,000 pounds.

David (Tripp Lite) The main AC unit for the server room are on the building circuit. The smaller backup AC unit is on the IT circuit. We have tested the backup unit and it won't keep the room at 66 which is where we keep the main unit set, but we haven't registered above 68 even on the smaller backup unit.

Nice. As a reference, I'd be interested to know how many power issues you saw before the redesign and how many you are seeing now. That is, I know you mentioned "
before we built the server room we had to make a lot of trips in due to power issues.
" and now you seem to be in a pretty good place. Did you used to have 99.5% uptime and you went to 99.8% or used to have to deploy service 10 times a year, now it's once, etc?

David (Tripp Lite)​ I've only been here 14 months, but I worked for the MSP that supported them the prior 15 years, so I know the history pretty well. Before the server room was build & I helped with the design. Someone would have to come address a power outage every 30 - 45 days. In the past 14 months, we haven't had to come in due to any to address power issues. With sites, Eastern to Pacific, and 2nd shifts, we don't have the luxury of waiting till tomorrow if something hiccups.

David (Tripp Lite)​ I've only been here 14 months, but I worked for the MSP that supported them the prior 15 years, so I know the history pretty well. Before the server room was build & I helped with the design. Someone would have to come address a power outage every 30 - 45 days. In the past 14 months, we haven't had to come in due to any to address power issues. With sites, Eastern to Pacific, and 2nd shifts, we don't have the luxury of waiting till tomorrow if something hiccups.

We do manage to only have to staff help desk 10 hours a day though.

Thanks mikehoneycutt​

It's always nice to get real-world feedback like this that show infrastructure upgrades can have an impact.