The power supply for the entire building (which houses several data centers, several web hosts, and some notable large sites like myspace.com) is completely redundant with incredible amounts of UPS power and 5 huge diesel generators. On paper it sounds very good.

During the power outage last year the UPS system worked but the generators did not. That was very bad but we decided to give them another chance. We have a lot invested in our current setup and changing locations is a very big undertaking. The power problems are affecting all of downtown Los Angeles so we would have to move to a completely other city. Many of our senior technical people live near our current office and data center and they would all end up moving themselves or enduring painful commutes in LA traffic. These are not excuses.. just some insight into the reality of the situation for us.

We have been with this building for several years and at the time of the first power outage that was the first time they had failed us. In hindsight, we should have had less faith in them. That’s very easy to see now.

For this second power outage of the local grid, the building’s UPS system failed but the generators did work. 0 for 2. Our building had been on generator power for the entire week since the local power grid had gone out and that was working fine until some sort of electrical malfunction took the generator system down yesterday afternoon. They quickly switched us back to grid power which is how things are now.

In a network like ours, it’s not just a matter of ‘rebooting a server’ as some of you have indicated. We have all user data stored on network file servers for redundancy and to optimize usage, for instance. After yesterday’s power blip all of the 68 file servers but one came up properly. That one that did not resulted in some websites being down for an additional hour and a half or so while we diagnosed the problem. We have now corrected the configuration problem and expect all file servers to come up normally in the future.

Obviously, loss of power to our data center is something we would rather never see happen. We are already in the process of acquiring some UPS equipment of our own for our most critical equipment and we will be evaluating all possible options including completely moving our data center operation to another location.

This entry was posted
on Saturday, July 29th, 2006 at 15:58 and is filed under DreamHost.
You can follow any responses to this entry through the RSS 2.0 feed.
Both comments and pings are currently closed.

4 Responses to “DreamHost Power Outage”

For what it’s worth, I’ve been considering DreamHost now for some type to provide my company with a dedicated server. After this, and last years downtime – I simply can no longer consider DH as a viable option.

It’s a shame, b/c I very much like DH and the server options they provide (as well as providing managed up-2-date software/services).