If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

Firmware issue stories

08-22-2013, 04:31 PM

I have been maintaining quite a few computers up to now, and have seen some really weird things happen with regards to their firmware (BIOS/UEFI). I'm wondering if I'm just unlucky, or if really that many people hit issues like that. So I'm interested to know what firmware issues you have run into so far on your computers.

In my case, the weirdest issues I've hit on a PC from 2007. It has a regular BIOS, the motherboard is by Gigabyte. In theory, it supports USB 2.0 devices. By default, however, legacy USB devices are turned off. Which means that to activate the support in the first place, you absolutely need a PS/2 keyboard. You would also need it if you restored BIOS defaults, either by clearing the CMOS or updating the BIOS.

But that's the smallest issue. When this support is enabled, the BIOS does discover USB drives that are connected, and it's possible to tell the BIOS to boot off them. However, that doesn't actually work as expected... If I try to boot anything complex, like a LiveCD, the BIOS just sits there, forever, not even getting to the point of showing SYSLINUX, and without any messages about it. And if I try to boot anything simple, like memtest86+... It fails so much it reboots. If the USB device is set to be primary boot media, that means an infinite restarting loop!

I thought that updating the BIOS version could probably help with this issue. So I downloaded the update, wrote it on a USB drive, then tried to get the BIOS to update itself. And even though it showed the drive and allowed to access the files... it still wouldn't accept the update file. And it was definitely the correct version. So I had to use @BIOS (would have tried FlashROM first, but, you know, USB boot didn't work). And, well, @BIOS is supposed to get the BIOS version, connect to Gigabyte's servers, download an update, then apply it (in the process possibly destroying the system, as @BIOS is known for its notorious unreliability). But it wouldn't connect to any Gigabyte server at all. So I had to manually specify the update file (the same I had written on USB). Thankfully, it worked fine, @BIOS accepted it and flashed it successfully.

That helped... a bit. It no longer hangs or reboots, but booting off anything over USB is still super slow. And, of course, there is no hope for any updated BIOSs for that motherboard at this point, as it's no longer maintained (the update was from 2009).

The next to worst issues I've hit was on an Asus EeePC with an AMD APU, using UEFI. It's one of those silent UEFIs that masquerade as BIOSs (so that consumers wouldn't discover what type of firmware it's running, perhaps? ). By default it does MBR boots through its legacy BIOS support, and there is no way to turn it off. I wanted to install Gentoo on it, and using all the capabilities, including UEFI and GPT. In order to install anything in UEFI mode, you need to boot into UEFI mode first. So I grabbed a Kubuntu Live USB. Thankfully the UEFI automatically tries to launch whatever is in EFI/BOOT/BOOTX64.EFI on USB media (an amazing feat for it, because it surely doesn't try to do that when booting from the hard drive itself!). The installation went fairly well (especially considering that Kubuntu isn't exactly an official Gentoo installation medium - but it sure is much easier to install from it than the official one!), until I got to the bootloader installation part. Now normally that should be easy, you just use grub2-install, tell it that it should use UEFI capabilities, then point it at the EFI system partition. Except that whenever I did that... the kernel would panic citing something about EFI variables, and on reboot nothing would be booted (the GRUB entry isn't added to the UEFI). I heard that different kernel versions would have a different effect, so I would use Kubuntu's GRUB to chainload the hard disk GRUB, then change a kernel version and try again. A slightly newer kernel didn't crash any more, but instead efibootmgr game me error messages and still wouldn't put an entry into UEFI. Updating to the latest kernel would make the kernel panic return.

So that was unfortunate, conventional means are apparently not enough to make the UEFI boot Linux natively. My first thought was to upgrade the UEFI. The Asus firmware update page is extremely confusing, though. The UEFI wouldn't upgrade to what Asus claims to be the correct firmware to use, but accepts the one that they claim to be for slightly different systems fine. Either way, an update changed nothing, the same issues stayed.

My last option was to add an entry manually through the UEFI shell. Problem? The UEFI doesn't include one. Instead they have an UEFI entry that boots from the first ShellX64.EFI file it finds (thankfully it searches the hard drive as well). On one hand, that could be a workaround ? the UEFI doesn't do any checks of what the file is, so naming GRUB "ShellX64.EFI" would make it boot. But there is no way to set that as the default option, so each time you wanted to boot, you'd have to enter the UEFI options, which is of course not an option. The real fix would be to boot into an actual UEFI shell and change the variables from there.

Since there is no official UEFI shell included anywhere, I had to use TianoCore's UEFI shell. It comes in two varieties ? UEFI shell v2 (for UEFIs compliant with UEFI 2.3+ spec) and UEFI shell v1 (for earlier ones). I tried v2 first, but, surprise surprise, it failed to boot. Apparently that UEFI is not UEFI 2.3 compliant. So I booted into shell v1... but it doesn't include bcfg, which is the command that actually does the UEFI variable changing. Thankfully the Arch wiki also has a link to a modified shell v1 with bcfg that finally allowed me to add a GRUB entry manually, and set it to default. Phew! Now I just hope nothing ever touches the EFI variables of that thing again (or else get an UEFI-induced kernel panic).

And that's just two of the firmwares I've had trouble with. I'll add my experiences with some of the others a bit later. In the mean while, does anyone else have firmware-related stories to share?

Basically you use bcfg to make a grub.efi boot entry. But that's a very inefficient way to do so. You can directly use efibootmgr instead - that way you can even use "-u '<your options>'" too, as the linux kernel can be build with efi stub you could use it directly with root option. Another trick is to copy grub or gummiboot at positions where a ms bootloader would be, basically there are only 2 types of firmware, one search em at "EFI/Boot/bootx64.efi" (generic location also used for removeable drives) but many use "EFI/Microsoft/Boot/bootmgfw.efi". If that file is there you usually see an entry called "Windows Boot Manager" in your setup routine. This feature was added to make it possible to swap the harddisk and boot from that - otherwise you would need an efi shell or grub (can search for efi binaries as well) or something similar to start what is on your hd. As /boot/grub/custom.cfg example you could use:

Comment

Basically you use bcfg to make a grub.efi boot entry. But that's a very inefficient way to do so. You can directly use efibootmgr instead - that way you can even use "-u '<your options>'" too, as the linux kernel can be build with efi stub you could use it directly with root option. Another trick is to copy grub or gummiboot at positions where a ms bootloader would be, basically there are only 2 types of firmware, one search em at "EFI/Boot/bootx64.efi" (generic location also used for removeable drives) but many use "EFI/Microsoft/Boot/bootmgfw.efi".

That's the point, that UEFI is so broken that whenever I launch efibootmgr, it makes the kernel panic. It's something about UEFI variable access that just doesn't work right there. On non-broken platforms it does work fine, yes. The BootX64.EFI doesn't work on the local hard drive. As for bootmgfw, I didn't try that, but it would make so sense if adding that would make an UEFI entry spontaneously appear. bcfg didn't list any other entries than I could see, and since the netbook came with MBR partitions, nobody expected people to try and install UEFI-aware Windows on it anyway.

So do you have similar stories to share, or did all the computers you use have sane firmware after all?

I can also remember the issues I had with the UEFI of my main PC here. The motherboard is really quite new, but the UEFI wasn't properly tested before, I assume, because trying to install Windows ? yes, not even talking about Linux yet ? in UEFI mode would cause the setup to instantly BSOD. Since I'm using a dual-boot solution here, that made me shelve the idea of making my main PC legacy-free for that moment (because Windows stupidly doesn't install into GPT disks if started in non-UEFI mode, although GPT and UEFI aren't actually strictly related). I reported the issue to MSI, and now after Windows 8 got released, they updated the firmware. There was nothing about it in the release notes, but updating the firmware did make Windows install correctly. Linux installation went smoothly (openSUSE makes it super easy to make UEFI installs, too). But after a while, the UEFI started running slowly, and one day it stopped being able to boot to UEFI GRUB. I upgraded the firmware again and it solved the issue once again (it was probably some form of corruption). I did end up having to reinstall UEFI GRUB to get it to work again, but again that's pretty easy to do through YaST. Aside from these issues, the UEFI on this motherboard is surprisingly sane. It includes an UEFI shell out of the box as part of the UEFI itself, I can change the boot order with ease there, and the latest update also allows me to switch off BIOS emulation, making the PC legacy-free.

Another UEFI that was a bit annoying is on my tablet PC. Though I didn't try to set up UEFI boot on it (it's a 32-bit device and it needs Windows due to the proprietary screen handling drivers, while 32-bit Windows don't support UEFI boot at all). However, whenever I tried booting Linux on it, I wouldn't be able to get the graphics running correctly, unless I passed ACPI_OSI="Linux" or disabled ACPI altogether. That UEFI is also one of those that masquerade as a BIOS, and it's so large that it didn't fit in the flash memory ? it uses some of the hard disk space, then masks it (which Linux sees right through, of course). But since that device uses an SSD that is soldered on anyway, it doesn't matter that much.

So that leaves about 2 BIOSs that actually performed without issues as fas as I can remember. One is on a laptop, the other one was on my previous PC motherboard that I now relegated to an HTPC (although its boot up speed is horrible when selecting AHCI boot for some reason).

Comment

I mainly test uefi since april 2012. At that time i added some entries with efibootmgr with my P8Z68-V board, and they booted fine. Then i exchanged the order inside the normal setup routine and the board did not even post at all. I should have send the board to repair completely but i only wanted a firmware chip, after 1 week writing emails to Asus support i got what i wanted. A few updates later they introduced a new firmware bug, that deletes entries - this is in every new board since then. If you know efibootmgr you should understand the problem:

As the efi partition is on fat you can change the case of the entries, so now i modify this to prevent em from being cleaned up, als they just leave the last entry created even if the options used are different.

Another board with an older UEFI implementaion (2.00 instead of 2.31) had the issue that when you formatted the fat partition directly with EFI as label then it did not find the EFI directory as it does not check if the attribute of that entry is directory - and the volume label entry was before that. That's the reason why the EFI partition inside the Kanotix iso images is called KANOTIX_EFI - we used EFI first but it did not boot on all boxes.

Also there are boards that clean up EVERY entry when you select the garbage collect option. Even if the entries are valid they are deleted - if they are not for starting Win

Comment

Interesting, I've never seen the clean up functionality anywhere. It sort of makes sense, but yea, that's kind of a magnet for bugs. I can understand how the logic of leaving in only the last entry makes sense: "Bah, nobody uses dual-boot anyway, so let's just delete all the unneeded boot entries left by previous installations every boot!" Except that people do use dual boot a lot, of course With deleting even valid entries, I can see another kind of insane troll logic at work: "Bah, nobody uses anything else than Windows anyway, and it overwrites the last bootloader when new versions are installed, so we can safely delete everything but the Windows bootloader option!" Except that it defeats the purpose of such an option in the first place!

As for the volume name, that's interesting, what happens when the volume doesn't have a label to begin with? I always set the label to "EFI SYSTEM" and it has always worked for me, but I never tried "EFI" or nothing at all. Actually, does Windows label the EFI system partition, or does it leave it empty?

Comment

Win hides the partition and does not use a volume label. The name does not matter, just EFI when set directly at format time did not work. When you "rename" it then it is usally after the EFI directory and it would work with a problematic system (had to look at the entries in hex mode) . but it is only optical anyway. Did you upgrade the firmware with your problematic system? I always use the latest firmware.

Comment

Win hides the partition and does not use a volume label. The name does not matter, just EFI when set directly at format time did not work. When you "rename" it then it is usally after the EFI directory and it would work with a problematic system (had to look at the entries in hex mode) . but it is only optical anyway. Did you upgrade the firmware with your problematic system? I always use the latest firmware.

Ah, thought so. Hopefully nobody will have the bright idea to only consider hidden partitions as the EFI system partitions

Yea, of course, when I hit an issue I always try to upgrade the firmware. Just that often times it doesn't do anything, or is already at the latest version.