Power Outage in Area where most openSUSE Servers are Located

Just a quick note: We have a power outage in the part of the city of NÃ¼rnberg where the Novell office and the main server room is.Â This means that many of our servers are right down, especially the download redirector, the mailing lists, the openSUSE build service and users.opensuse.org.

I will post a message once the power has been restored and all machines are running again.Â Current estimate (11am Nuernberg time) is that it will take another 4 hours (until 3pm Nuernberg time which is 13:00 UTC) at least to restore power.

Note: the power companies do not know yet exactly where the problem is.

This server and the wiki are located in another data center and are therefore available.

Updates:

13:15 CEST: New rumor: Current estimate for power restoring is six more hours, they need to dig up the street.

16:45 CEST: Bad news: It will take longer until power gets restored.Â The local power company just stated “22:00 to 23:00”.Â We will try to get then the first machines up but might not get everything running during the night.Â Btw. currently it seems that it’s only our office complex that is without power, the rest of the area has power again.

17:15 CEST: I just chatted with our admins, and they currently hope to have everything up Saturday around 13:00 CEST (11:00 UTC) if – and only if – there are no major problems like hardware failures.

18:05 CEST: The admins will start early tomorrow morning – there’s no sense waiting for the power company this night.Â The estimate stays at 13:00 CEST (11:00 UTC).Â We’ve never experienced such a long outage before, this is exceptionally bad.

19:02 CEST: Beineri has uploaded some photos from the construction site (thanks!).

20:04 CEST: Marko has uploaded some photos as well (thanks!).Â Some notes: I’ve heard (no official confirmation) that our office building has two power lines and currently both are getting repaired, they started with the first one and now dig out the second one as well.Â Our building seems to be the last one in the area to get power back since it’s the only one with a 20kV line.

1:30 CEST: Power is back in the office – later than estimated.

9:20 CEST: Our admins have brought the basic net infrastructure up and will work on the rest now.

9:45 CEST: The first servers coming up, download.opensuse.org is available again.

10:20 CEST: lists.opensuse.org is up again, I’ve send an announcement out to the mailing lists.Â I just don’t know when it will go through since some other systems are not running and I guess the mail queue is rather long.

10:33 CEST: After I approved my announcement, it went through directly and was sent out – this means, the infrastructure is indeed up and runing ;)

13:00 CEST: Most systems should be up, the only problems right now are login on users.opensuse.org and the build service.

15:00 CEST: Info from our admins:

It has turned out that the electric feeder cable outside the building was blown which had to be digged out and then repaired, so the first estimationÂ of the energy provider was a little bit optimistic. Connection was re-established Friday night at about 1AM (localtime) and reconstruction started this morning at 7AM and most important services were back at about 9AM.

15:08 CEST: We’re still working on users.o.o and the build service, everything else should be ok.

18:50 CEST: users.opensuse.org and build.opensuse.org are back online.Â We should now be good enough for the weekend.Â Currently still down are ideas.o.o, features.o.o and tracker.opensuse.org (for our torrents).Â We will have these restored on monday.

123 Responses to “Power Outage in Area where most openSUSE Servers are Located”

It’s terrible than an error outside your company can disrupt nearly the complete Linux community. Sudden power outages CAN damage disc structures and even hardware components if there is not enough time (UPS backup power supplies can give grace time) to shut down the server computers gracefully.

I think this situation would bring the openSuSE community in a black hole, but a UPS system or a small and indipendent electric generator could probably help us all. Unfortunately servers gone down, so you have to raise those servers up as soon as possible, and as a strawberry on the cake as you could post the official openSuSE management office apologies. Thanks for this advice.

There is no apology needed, as there was no involvement by any of the Novell/SUSE staff. A generator or even cluster of generators would only provide at most 2hrs power to enable a graceful shutdown of all servers. You could possibly stretch an extra hour out with a cluster of UPSes. There is no alternative other than having your own power station ;)

As it happens the community by and large have rallied very well together and have been constructive in trying to help out. That is what is required, not negative sarcastic remarks :) One very good way would be to donate to unixheads.org

I must agree with Dean — our office has two Natural Gas generators, 39KW and 45KW. Our small office only has 35 employees and 8 servers, but we can go as long as we need, just as long as we get natural gas supplied to our generators.

I’m somewhat surprised that such a critical data center for Novell wouldn’t have something like this.

I think you’ve forgotten that this is openSuSE this is a sponsored opensource distro. As openSuSE is not the core activity of Novell they won’t have huge amounts of money to have generators and/or a redundant set-up in two or more locations.

I think the people @ openSuSE are doing a great job as it is. This is probably also quite nightmare to them.

Maybe a quick win, in case something like this happens again. That’s a download.opensuse.org. I’ve understood that this part of openSuSE is mostly about redirecting downloads. Therefore it doesn’t need a lot of processing power. By putting a second download.opensuse.org server in the second location and distributing a routes via BGP with different costs to the internet the second download.opensuse.org would have done the job now. Than most people won’t even notice such an outage.

Hmm. Is it really necessary to complain for something completely independent and out of Novell/openSUSE control? As a community, let’s be supportive!

Asking for apologies seems a bit too much for me, and no openSUSE community won’t go into a black hole for a blackout. We are a lot stronger that what you think. And to put it fair, Novell gives us openSUSE for free, it supports it for two years and if the redirector, not the mirrors, are down for some hours, well, I think we can be happy anyway. Just take a break and go for a walk. Your life will improve!

What a fine surprise this is! I decided to convert one of my servers to openSuSE tonight (from Fedora which was running just fine) and have spent the better part of 3-4 hours trying to figure out what I did wrong in the install! :-)

Had a bad DVD so was going with the Live CD install. Nice to know I’m not quite as dense as I was beginning to think.

Have one question for anyone. When I run with the NetworkManager, it consistently loses the nameserver in /etc/resolv.conf and I have to open another terminal session to add the nameserver 192.168.1.1 to get to my router. Anyone know what I’m configuring wrong in the network setup with Yast?

I live just 400 meters away, and power never went down here (I would have noticed). But you can tell how the city is divided into grids for power supply: Just walk 100 meters towards the SUSE office, and even the traffic lights are dead. It’s like in Sim City when Godzilla stomped out a couple of city squares. ;-)

I was just installing 11.0 and yast kept refusing to update. I couldn’t mind to check the downloads page and instead installed the whole system over and over again. :D after the 6 th instllation I’ve checked downloads server and then saw this news. Hope we get servers soon.

My company is located in the same Building where Suse is located and my company is closed because of the outage.
It’s really terrible, all Buildings around us have power, only the linux business campus has no power.

Actually a backup generator can provide power for several days… It is all relevant to how much money you want to invest into it. I have 4 seperate equipment sites all of which have backup generators that can supply power for several weeks before they run out of fuel. I am not talking small sites either since at least one site has 1000kw diesel generator. Combine the generator with a UPS to cover the spin up time of the generator and a auto switch and you will have seemless power even during extended power outages like your having now.

If I were Novell I would invest as much on generators as I invested in openSuse :-/
Seriously guys, are we really giving out because something that is free is temporarily unavailable. Its not even as if our os’s are affected, and any software you might need urgently can be found elsewhere with nearly no effort.

I really didn’t realise just how much i mess around with OpenSUSE on my laptop until all the repro’s have gone down. Hope this get back to normal soon and i know the team must be working hard to get things back to normal for us.

It’s a conspiracy, if I hadn’t set my home page to opensuse news I would have been banging my head against the keyboard, no doubt for hours.
Good luck everyone, I guess I’ll get my updates some other time. It is Friday, I believe it is time to have a beer.cheers.

Interesting pictures. No hard hat. This would not be allowed over here. I wonder if the the guys even wore safety boots.
Keep up the good work SuSE gang. I will be in your general neighbourhood next week for a couple of weeks.

I would point out to all of the people with the idea of adding a generator. Generators are great, and sure, perhaps Novell should purchase one. BUT, please note the BUT, if the power outage shutdown any of the T1 type circuits, which also may be powered in that region, it really doesn’t matter if you have a generator or not. You won’t have connectivity. The only sound solution is to have a mirrored system with alternates in the DNS. I want to thank the open suse staff for all of the fine work they do and for keeping us up-to-date on the power issues.

Best to jump to a date, a beer, a good weekend,
time to think around, it is just like holidays!

I would like to thank people at OpenSUSE for the
great software they are providing us for free! I
work as Network and Operating Systems Analist for
long time, and Linux is my basic tool for more than
10 years. After I came to know SuSE and then OpenSUSE,
I used it and I am not a little bit disappointed with
the results.

So, an outage that is NOT my fault and is NOT any
OpenSUSE people, well, those things just happen! I
can see NO reason being angry or the like. Lets take
advantage of that as we can.

Worst part when it happened was that their prefix in BGP routing table went down and they are not routable any more…
I called my ISP to shout on them why do not they forward that routes to me, and just than i went to homepage to see about problems with electricity :) :)

Anyway quick fix if you really need to download stuff trough YAST is (adjust it a bit for accuracy and not to saturate mirrors!!)
cd /etc/zypp
cp -a repos.d repos.d_backup-20081010
cd repos.d/
for i in *.repo ;do sed -i -e "s*http://download.opensuse.org/repositories/*http://ftp5.gwdg.de/pub/linux/suse/opensuse/repositories/*g" "$i";done
for i in *.repo ;do sed -i -e "s*http://download.opensuse.org/update/*http://ftp5.gwdg.de/pub/linux/suse/opensuse/update/*g" "$i";done
for i in *.repo ;do sed -i -e "s*http://download.opensuse.org/distribution/*http://ftp5.gwdg.de/pub/linux/suse/opensuse/distribution/*g" "$i";done
you

and yes i know i could make it in one line but with this I am sure no unneeded update is performed

These things happen, but sadly it is easier for people to criticize and be negative than supportive. It is great to see so much of the community are supportive and understand the nature of the beast that is power cuts. Just sad that the odd few have to be negative.

My thanks to the people who will be working hard and have worked hard to be able to bring OpenSUSE to us.

Is it Murphy’s birthday or what?! I drove 52 miles to do this installation, and sure enough – BLAM!

I have THREE DNS servers I am reimaging to SuSE 11, and I stupidly popped in LiveCD disks in all of them at the same time. Yes, I do have two other DNS servers handling the load, but they’re not idle that’s for sure!

I blissfully started the reimage, and formatted out partitions, reconfigured IPs, DNS servers, etc, and sure enough I keep getting http://download.opensuse.org/xxxxx not found – I thought I screwed up the NIC configurations, because all three kept failing. I thought, OK, something is whack in the SuSEFirewall, so I disabled it too

I am trying to convince my associate to mirror host download.opensuse.org – to give back to the community.

If there’s a moderator reading this, please forward off my info to the persons in charge, so we can discuss equipment and performance needs to play mirror. He has rackspace, and bandwidth, especially for package updates.

Also, if anyone knows of a different repository I should point to so I can finish these updates, please let me know! b.kinney1…A..T..comcastDOT..net (you get the idea)

In any case, a big thank you to everyone who keeps this set of services running – I personally know that you are all doing the best you can under the circumstances.