Archive for October, 2008

openSUSE 11.1 beta 3 will be delayed by several days. We originally scheduled the release for today (October 16) but the power outage last Friday left us unable to check in packages temporarily, and pushed the schedule back by several days.

We had a first build this morning and have checked in more than 20 packages to fix the most serious problems. We expect to release the third beta on Monday (October 20), or later, depending on the status of the builds on Friday.

We’ll give an update Friday as to the status of the next beta release. Thanks for your patience, and we look forward to your help in testing and working on openSUSE 11.1 when we release beta 3!

As it stands now, over 57% of votes have been cast. Hopefully the extension will enable all voters the chance to have their voice heard. Remember, this is the community’s chance to influence the governance of the openSUSE project. If you don’t vote your chance to influence the project will be reduced. If you have questions about the candidates, ask them. They won’t bite, but they may be able to give you the answers to enable you to vote with a clearer consience.

The patches we did for the Intel e1000e network card for Beta2 protect the chip so that the NVRAM could not get corrupted anymore and we indeed did not receive any new bug reports and could not reproduce the bug anymore on our systems.

Further investigation by Intel has found the root cause of the problem as Steven Rostedt wrote on the linux kernel mailing list : The dynamic ftrace code contained some fragile code that could write to ioremap-ed memory and thus corrupt the NVRAM.Â The issue could happen “when the init functions of a module are freed and the nvram is vmapped there as well”.Â The full story can be found on LKML.

Since 24th of September, we have disabled for our kernel of the day the dynamic ftrace code due for all flavors except the debug and vanilla kernels (on x86 and x86-64 – it was not enabled on other architectures).Â We have also added the NVRAM protection patches to all kernel flavors.Â Therefore Beta2 already contains – by pure luck ;) – not only the NVRAM protection but also not anymore the broken code.

Beta3 will contain the same fixes – and the kernel of the day has just been updated with dynamic ftrace code disabled also for the debug and vanilla kernels (with the update to 2.6.27.1).

So, if you’re running a debug or vanilla kernel, I advice – to be on the safe side – to update to the 2.6.27.1 kernel of the day.Â For everybody else: The Beta2 and Beta3 kernels should not corrupt your Intel e1000e NVRAM.

I’d like to thank all that were involved in debugging and fixing the issues around this, including our kernel developers Karsten Keil and Jiri Kosina who debugged and worked on a solution, testers that fried their machine and helped debugging like Stephan Binner and Vladimir Botka, and the team at Intel for developing protection code and finding and fixing the root cause.

Update 2008-10-21: Beta3 will contain 2.6.271.1

Fixing Erased e1000e NICs

Karsten Keil has developed a way to fix broken e1000e eproms.Â Please contact him at kkeil@suse.de in case you need to recover from this bug.

After very careful deliberation, we’ve chosen the winners from Hack Week III. The categories were: Best Cross-Pollination Team, First Penguin Awards, and the Best Overall Projects.

The Best Cross-Pollination Team: Awards the project that connects different teams, and raises awareness of each other’s work.

First Penguin Award: Awards the project that goes out on a limb to try something difficult and risky… and probably fails.Â Named in honor of the first penguin to jump into the water when there may be predators swimming below.

Best Overall Projects: Goes to the projects that the panelists thought were â€œbestâ€ for one reason or other. Surprisingly enough, we had a tie for first place on this one!

Just a quick note: We have a power outage in the part of the city of NÃ¼rnberg where the Novell office and the main server room is.Â This means that many of our servers are right down, especially the download redirector, the mailing lists, the openSUSE build service and users.opensuse.org.

I will post a message once the power has been restored and all machines are running again.Â Current estimate (11am Nuernberg time) is that it will take another 4 hours (until 3pm Nuernberg time which is 13:00 UTC) at least to restore power.

Note: the power companies do not know yet exactly where the problem is.

This server and the wiki are located in another data center and are therefore available.

Updates:

13:15 CEST: New rumor: Current estimate for power restoring is six more hours, they need to dig up the street.

16:45 CEST: Bad news: It will take longer until power gets restored.Â The local power company just stated “22:00 to 23:00″.Â We will try to get then the first machines up but might not get everything running during the night.Â Btw. currently it seems that it’s only our office complex that is without power, the rest of the area has power again.

17:15 CEST: I just chatted with our admins, and they currently hope to have everything up Saturday around 13:00 CEST (11:00 UTC) if – and only if – there are no major problems like hardware failures.

18:05 CEST: The admins will start early tomorrow morning – there’s no sense waiting for the power company this night.Â The estimate stays at 13:00 CEST (11:00 UTC).Â We’ve never experienced such a long outage before, this is exceptionally bad.

19:02 CEST: Beineri has uploaded some photos from the construction site (thanks!).

20:04 CEST: Marko has uploaded some photos as well (thanks!).Â Some notes: I’ve heard (no official confirmation) that our office building has two power lines and currently both are getting repaired, they started with the first one and now dig out the second one as well.Â Our building seems to be the last one in the area to get power back since it’s the only one with a 20kV line.

1:30 CEST: Power is back in the office – later than estimated.

9:20 CEST: Our admins have brought the basic net infrastructure up and will work on the rest now.

9:45 CEST: The first servers coming up, download.opensuse.org is available again.

10:20 CEST: lists.opensuse.org is up again, I’ve send an announcement out to the mailing lists.Â I just don’t know when it will go through since some other systems are not running and I guess the mail queue is rather long.

10:33 CEST: After I approved my announcement, it went through directly and was sent out – this means, the infrastructure is indeed up and runing ;)

13:00 CEST: Most systems should be up, the only problems right now are login on users.opensuse.org and the build service.

15:00 CEST: Info from our admins:

It has turned out that the electric feeder cable outside the building was blown which had to be digged out and then repaired, so the first estimationÂ of the energy provider was a little bit optimistic. Connection was re-established Friday night at about 1AM (localtime) and reconstruction started this morning at 7AM and most important services were back at about 9AM.

15:08 CEST: We’re still working on users.o.o and the build service, everything else should be ok.

18:50 CEST: users.opensuse.org and build.opensuse.org are back online.Â We should now be good enough for the weekend.Â Currently still down are ideas.o.o, features.o.o and tracker.opensuse.org (for our torrents).Â We will have these restored on monday.

The openSUSE Project is going to have a presence at the Indiana Linuxfest and Ohio Linuxfests this weekend.

Friday, the Indiana Linuxfest is taking place in Bloomington, Indiana at Indiana University in the Indiana Memorial Union from 10 a.m. to 4 p.m. openSUSE Community Manager Joe ‘Zonker’ Brockmeier will be giving the opening keynote, “Building Community and Taking Linux to the Masses,” at 10 a.m.

The Indiana Linuxfest has a diverse schedule, including talks on iPhone development, enterprise virtualization, and Sun’s open source efforts.

The Ohio LinuxFest 2008 takes place Friday and Saturday. Friday is Ohio LinuxFest University, and Saturday is the main day of the ‘fest, including talks, exhibits, and a afterparty that’s not to be missed.

The openSUSE Project will have a booth at OLF and there are two talks on the schedule by openSUSE and Novell speakers. Joe ‘Zonker’ Brockmeier will be giving the 8 a.m. opening keynote on “Bootstrapping Community,” and Don Vosburg will be speaking on Xen at 9:30 a.m.

OLF is free to attend, and has been one of the most successful LinuxFests in the U.S. for several years running. It’s not too late to register, or just show up early Saturday morning.

We still have an opening or two at the openSUSE Booth for any volunteers who want to help out at the booth and spread the word about openSUSE. If you’re interested, just drop a note to Zonker.

The first beta release for 11.1 was so popular, we’ve decided to do it again! The openSUSE Project is happy to announce the release of openSUSE 11.1 beta 2, available for immediate download and testing.

Please note:
* LiveCDs are being regenerated and released on Monday (Oct. 6th).
* Also, the x86_64 Beta1-Beta2 delta.iso is currently broken — to be released on Monday as well.
* the new factory tree is available from http://download.opensuse.org/factory/repo/oss/.

This is an update on the status of the e1000e issue. Our openSUSE 11.1 Beta 1 release contained a bug that would cause the non-volatile memory (NVM) of the e1000e controller to be corrupted in certain hardware combinations. This NVM is shared with other components of the system.

We are still working on root-causing the issue, in close cooperation with developers from Intel and the general Linux community.

In the Beta 2 release, the e1000e driver has been augmented with several kernel patches that prevent all of the plausible scenarios where the NVM would be overwritten. This includes a patch that enables write-protection of the NVM. In order to corrupt the NVM with this patch in place, an application or kernel module would have to undo this write protection explicitly before being able to erase the NVM. Most of these protective measures have been implemented within the e1000e driver.

All tests performed so far have shown that with these protections in place, we are unable to reproduce the NVM corruption that could be seen with beta1. On some machines that were tested, a beta1 installation could lead to NVM corruption within 10-30 reboots, whereas a beta2 installation would perform 270 reboots without corrupting the NVM.

Therefore, we have made a conscious decision to leave the e1000e driver enabled by default. We think with the additional safeguards in place (most of which are part of the e1000e driver), the NVM is better protected than without loading the driver. This is based on the assumption that the e1000e driver shares the NVM with other parts of the system.

Nevertheless, users have the option to install beta2 with the e1000e driver disabled, by adding the following to the kernel command line when booting from the installation CD/DVD:

broken_modules=e1000e

This will prevent the driver from being loaded during installation, and will also add it to the file /etc/modprobe.d/blacklist, which will prevent it from being loaded automatically in the future.