Ubuntu boot problems

Ben Okopnik [ben at linuxgazette.net]

Mon, 16 Nov 2009 18:49:55 -0500

Earlier today, I had a problem with my almost-brand-new Ubuntu 9.10
install: my netbook suddenly stopped booting. Going into "rescue" mode
showed that it was failing to mount the root device - to be precise, it
couldn't mount /dev/disk/by-uuid/cd4efbe9-9731-40a5-9878-06deff19af06
(normally, a link to "/dev/sda1") on '/'. When the system finally timed
out and dropped me into the "initramfs" environment/prompt, I did an
"ls" of /dev/disk/by-uuid - which showed that the link didn't exist.
Yoiks... had my hard drive failed???

I was quickly comforted by the existence of another device in there, one
that pointed to /dev/sda5, a swap partition on the same drive, but that
could still have meant damage to the partition table. I tried a few
things, none of which worked (i.e., rebooting the system would always
bring me back to the same point)... until I decided to create the
appropriate symlink in the above directory - i.e.,

and exit "initramfs". The system locked up when it tried to boot
further, but when I rebooted, it gave me a login console; I remounted
'/' as 'read/write', then ran "dpkg-reconfigure -plow ubuntu-desktop",
the "one-size-fits-all" solution for Ubuntu "GUI fails to start"
problems, and the problem was over.

NOW, Cometh The REALLY Big Problem.

1) What would cause a device in /dev/disk/by-uuid to disappear? Frankly,
the fact that it did scares the bejeebers out of me. That shouldn't
happen randomly - and the only system-related stuff that I did this
morning was installing a few packages (several flavors of window manager
so I could experiment with the Ubuntu WM switcher.) I had also rebooted
the system a number of times this morning (shutting it down so I could
take it off the boat and to my favorite coffee shop, etc.) so I knew
that it was fine up until then.

3) What would be an actual solution to this kind of thing? My approach
was based on knowing about the boot process, etc., but it was part
guesswork, part magical hand-waving, and a huge amount of luck. I
still don't know what really happened, or what actually fixed the
problem.

> 1) What would cause a device in /dev/disk/by-uuid to disappear? Frankly,
> the fact that it did scares the bejeebers out of me.

AIUI this link is created by "udev" on the Ubuntu system so your
experience is truly difficult to understand. The initrd does not even
contain a "/dev" directory --- the initrd scripts creates it and the
relevant nodes when they run.

Under "udev" the /dev is a ram file system. In particular, the init
script initrd:/scripts/init-bottom/udev moves the /dev from the
initramfs to the real/final file system!

I can't see much of a solution. Here are some magical incantations
that may help you diagnose/fix the problem.
1. This should backup your initramfs and and create a new one.
You can then compare the two.

update-initramfs -u

2. To examine what there is in your initrd you can do:

mkdir /tmp/initrd
zcat /boot/initrd*img | cpio -idumv
cd /tmp/initrd

3. Check the output of "blkid" and compare it with "/etc/blkid.tab".
Also check this with the entry that "grub" has for the kernel
command line. This may provide some hints of whether the block
device id's are somehow incorrectly specified.

> Hello,
>
> On Mon, 16 Nov 2009, Ben Okopnik wrote:
> > 1) What would cause a device in /dev/disk/by-uuid to disappear? Frankly,
> > the fact that it did scares the bejeebers out of me.
>
> AIUI this link is created by "udev" on the Ubuntu system so your
> experience is truly difficult to understand. The initrd does not even
> contain a "/dev" directory --- the initrd scripts creates it and the
> relevant nodes when they run.

That's exactly why I was so annoyed by this whole thing. Why wouldn't
the scripts create that UUID link? Particularly since they did create
/dev/sda1, etc.?

It bugs me because it a) violates my understanding of how the "initrd"
process works, which I thought was pretty solid, and b) because the one
thing I rely on to tell me what happened, the system logs, don't exist
in an "initrd" environment. I know, I know - I'm asking for too damn
much - but without logs, there's no way to do any forensics... and so no
way to know what actually happened.

Oh, nice! Thank you, Kapil - I hadn't thought of that. Although as I see
it, the process of creating the UUID links is something that happens
when the "initrd" scripts run, and is not a "permanent" part of the
structure.

> 3. Check the output of "blkid" and compare it with "/etc/blkid.tab".

[blink] That's... interesting. It seems that my /etc/blkid.tab is a
broken link to /dev/.blkid.tab - and the latter doesn't exist.

Hm. This may be a clue, although I'm not quite sure of what the
mechanism would be.

> Also check this with the entry that "grub" has for the kernel
> command line. This may provide some hints of whether the block
> device id's are somehow incorrectly specified.

No, the device IDs are actually correct - I checked that first thing.
The onlty problem was that the /dev/disk/by-uuid/<UUID> for /dev/sda1
wasn't being created by the scripts, although the entry for /dev/sda5
was. When I plugged in a bootable USB thumbdrive and managed to boot to
a console, /dev/disk/by-uuid contained an entry for the USB media and
/dev/sda5 - and still no UUID link for /dev/sda1.

>> 3. Check the output of "blkid" and compare it with "/etc/blkid.tab".
>
> [blink] That's... interesting. It seems that my /etc/blkid.tab is a
> broken link to /dev/.blkid.tab - and the latter doesn't exist.
>
> Hm. This may be a clue, although I'm not quite sure of what the
> mechanism would be.

I just checked on my Ubuntu 9.10 setup and the /etc/blkid.tab file on my
system is also a broken link to /dev/.blkid.tab.

So I think that this is probably a separate issue 'cause I am not facing
the same problem as you. yet.

> Hi Ben,
>
> >>3. Check the output of "blkid" and compare it with "/etc/blkid.tab".
> >
> >[blink] That's... interesting. It seems that my /etc/blkid.tab is a
> >broken link to /dev/.blkid.tab - and the latter doesn't exist.
> >
> >Hm. This may be a clue, although I'm not quite sure of what the
> >mechanism would be.
>
> I just checked on my Ubuntu 9.10 setup and the /etc/blkid.tab file
> on my system is also a broken link to /dev/.blkid.tab.

OK, that's good data - thanks.

> So I think that this is probably a separate issue 'cause I am not
> facing the same problem as you. yet.

I'm just noodling here, but... what if it went something like this?

a) Something somewhere somehow [1] hoses the data in /etc/blkid.tab -
specifically, the data for the boot device. Since "blkid" (and more to
the point, libblkid(3)) uses the cached block device data in
/etc/blkid.tab, we now a problem - but it's not going to be visible
until the next boot (SURPRISE!!!)

b) The "initrd" scripts would normally "pivot_root" over to the boot
device, which would then run its own part of the boot sequence including
setting up the UUIDs for the devices and mounting everything via "mount
-a" - but that's not going to happen, since at least one boot-related
program depends on the info from 'libblkid' to figure out the devices.
In fact (and I had a bet with myself on this, which I won), here's the
list:

3) We now have a Catch-22 situation: the system "thinks" that we don't
need to detect the block devices, since we have an "/etc/blkid.tab",
etc.

So, the remaining question is, does "initrd" copy /etc/blkid.tab to the
/dev of the "full" environment? Did my creating the device and doing a
"mount -a" rewrite that file? Not having one (or having one that's a
broken symlink, I'd guess) causes 'libblkid' to actually re-read the
devices, according to the docs - so this works. Having a
borken or non-empty cache file may well be the "poison pill" that causes
the blowup.

If it ever happens again, I'll check that out. Perhaps, just for
experiment's sake, I should go ahead and create a "mostly-valid" cache
file and try rebooting...

[1] Does the exquisite precision of definition simply take your breath
away, or what?

> > 3. Check the output of "blkid" and compare it with "/etc/blkid.tab".
>
> [blink] That's... interesting. It seems that my /etc/blkid.tab is a
> broken link to /dev/.blkid.tab - and the latter doesn't exist.

That is certainly mysterious. While /etc/blkid.tab is indeed a link to
/dev/.blkid.tab, the latter file should exist.
(See /etc/blkid.conf for where this is configured)

Actually, I was wrong in my previous mail. The data in /etc/blkid.tab
is the cached data used while running '/sbin/blkid' so the output
of these is likely to be the same unless one uses '/sbin/blkid' to
regenerate the cache. So perhaps you should run

/sbin/blkid -w /tmp/blkid.tab

to see what file it will write and then write /etc/blkid.tab. It
looks like this for me:

If 'blkid' is managing to produce the correct output then you will
have to look deeper ... ;-)

You may want to look through /lib/udev/rules.d/60-persistent-storage.rules
which is the set of rules that creates the by-uuid links.

There are a number of cases in which the by-uuid rules seem to be
skipped (look for GOTO="persistent_storage_end"). Perhaps this will
help you to figure out some reason why the uuid link was not created.

> Hello,
>
> On Tue, 17 Nov 2009, Ben Okopnik wrote:
> > > 3. Check the output of "blkid" and compare it with "/etc/blkid.tab".
> >
> > [blink] That's... interesting. It seems that my /etc/blkid.tab is a
> > broken link to /dev/.blkid.tab - and the latter doesn't exist.
>
> That is certainly mysterious. While /etc/blkid.tab is indeed a link to
> /dev/.blkid.tab, the latter file should exist.
> (See /etc/blkid.conf for where this is configured)

This, to me, is looking like a bug in the whole "blkid" caching concept.

> Actually, I was wrong in my previous mail. The data in /etc/blkid.tab
> is the cached data used while running '/sbin/blkid' so the output
> of these is likely to be the same unless one uses '/sbin/blkid' to
> regenerate the cache. So perhaps you should run
> /sbin/blkid -w /tmp/blkid.tab
> to see what file it will write and then write /etc/blkid.tab. It
> looks like this for me:
> <device DEVNO="0x0802" TIME="nnnn" UUID="uuid1" TYPE="swap">/dev/sda2</device>
> <device DEVNO="0x0803" TIME="mmmm" LABEL="label1" UUID="uuid2" SEC_TYPE="ext2" TYPE="ext3">/dev/sda3</device>
> <device DEVNO="0x0804" TIME="llll" UUID="uuid3" TYPE="lvm">/dev/sda4</device>
>
> If 'blkid' is managing to produce the correct output then you will
> have to look deeper ... ;-)

In other words, it prints the output to the screen instead of writing it
to the specified file. This is contrary to what its man page says. I'll
be filing a bug in a minute.

> You may want to look through /lib/udev/rules.d/60-persistent-storage.rules
> which is the set of rules that creates the by-uuid links.
>
> There are a number of cases in which the by-uuid rules seem to be
> skipped (look for GOTO="persistent_storage_end"). Perhaps this will
> help you to figure out some reason why the uuid link was not created.

Actually, I've just managed to reproduce the initial problem I wrote
about - and, if I recall correctly, in the same way that I originally
created it. It still is a reportable bug against "blkid", and here's
what happened.

Shortly before the first crash, I was trying to create a bootable USB
stick for a friend (this is a surprisingly difficult task under 9.10.
No, "usb-creator" doesn't work. No, "unetbootin" doesn't work. No,
copying the CD contents plus running "syslinux" doesn't work. It
requires ugly "fdisk" hacking, plus copying, plus "syslinux -s", and
even then it only works sometimes.) Of course, in order to test this
thing, I had to reboot again and again... and at one point, I forgot to
set the BIOS to boot from the flash drive, at which point it started to
boot from my hard drive.

(Here's the denouement; drum roll, please.)

Being the impatient sort, I said "no, dammit!", and held down the power
button, aborting the boot half-way through. Now, I've done this before,
countless times, and didn't think anything of it... but the next time I
booted, the system couldn't find my /dev/sda1, or at least its UUID
link. Given the whole "blkid" caching scheme, it's easy to see why.

As to reproducing it - last night, right before going to bed, I was
trying to create a bootable thumbdrive for myself (I guess I just like
pain) - and repeated the exact sequence from above (forgot to set the
BIOS, interrupted the boot.) Again, I got the same crash half-way
through - but this time, because of the discussion we've had, I looked
at "/etc/blkid.tab". Bingo: it only contained the entry for "/dev/sda5"
(my swap partition), and none for /dev/sda1. I did a "cat /dev/null >
/etc/blkid.tab", rebooted, and got the console login prompt - not the
perfect outcome, but one easily solvable with "sudo dpkg-reconfigure
-plow ubuntu-desktop". One more reboot, and I'm now back into the GUI
environment and typing this email.

I've just checked "/etc/blkid.tab", by the way, and now it looks fine:

This seems to be a bug in the handling of the '-w' option. The
following alternate approach to generate the blkid.tab in an
different location works.

/sbin/blkid -c /tmp/blkid.tab

This creates the /tmp/blkid.tab

> at which point it started to boot from my hard drive.

> (Here's the denouement; drum roll, please.)

> Being the impatient sort, I said "no, dammit!", and held down the power
> button, aborting the boot half-way through. Now, I've done this before,
> countless times, and didn't think anything of it...

> > Again, this whole dance shouldn't be necessary. Relying on cached
> > device data _even when the system fails to boot_ is a bad, bad idea.
>
> I'm still mystified how this data or the lack of it is making it to
> the initrd. Some investigation is called for!

There is one more magic mantra that can be used to debug the Debian
(and hence Ubuntu) boot process. That is to give 'break=<value>'
option on the kernel command-line. (Don't forget to disable the
'splash' kernel option as well!).

Here <value> can be top, modules, premount, mount and init.

This will introduce a 'break-point' in the boot process at the
designated place.

Using this, I was able to interrupt the boot proces on my system at
the top to check whether there was any sign of the /etc/blkid.tab
file. There was not. I then put a break at premount to check whether
the /dev/disk/by-uuid links had been created. They had indeed!

AFAICS the initrd portion of the boot process does not use cached data
from /etc/blkid.tab.

Moreover, the '/dev' directory is a tmpfs which is 'move'd from the
initrd, so the links created during the initrd process should have
persisted on the live system.

Finally, creating a link in this filesystem from the initramfs should
have nothing to do with the link existing at the time of the next
boot!

> Hello,
>
> On Wed, 18 Nov 2009, Kapil Hari Paranjape wrote:
> > > Again, this whole dance shouldn't be necessary. Relying on cached
> > > device data _even when the system fails to boot_ is a bad, bad idea.
> >
> > I'm still mystified how this data or the lack of it is making it to
> > the initrd. Some investigation is called for!
>
> There is one more magic mantra that can be used to debug the Debian
> (and hence Ubuntu) boot process. That is to give 'break=<value>'
> option on the kernel command-line. (Don't forget to disable the
> 'splash' kernel option as well!).
>
> Here <value> can be top, modules, premount, mount and init.
>
> This will introduce a 'break-point' in the boot process at the
> designated place.

Kapil, you're now our official guru for the boot process.

> Using this, I was able to interrupt the boot proces on my system at
> the top to check whether there was any sign of the /etc/blkid.tab
> file. There was not. I then put a break at premount to check whether
> the /dev/disk/by-uuid links had been created. They had indeed!

So, somewhere in between those two, my system fails to create the UUID
link for /dev/sda1 when I hard-reboot it - although it clearly does
detect the device itself, and create the /dev/sda{1,5} entries for it.
Moreover, it creates the UUID link for /dev/sda5.

I don't know the mechanism of this part well enough to understand how or
why it would fail, though.

> AFAICS the initrd portion of the boot process does not use cached data
> from /etc/blkid.tab.
>
> Moreover, the '/dev' directory is a tmpfs which is 'move'd from the
> initrd, so the links created during the initrd process should have
> persisted on the live system.

Well, once booted, everything looks fine - but that's a sort of a moot
point, since it wouldn't have booted without it all being fine. The
problem seems to be that there's no graceful fallback when there's no
UUID entry. The closest thing to it is when (or if) you get a shell -
but then, you have to know where things failed, etc.

> Finally, creating a link in this filesystem from the initramfs should
> have nothing to do with the link existing at the time of the next
> boot!

The former didn't work - and I suspect the latter wouldn't have either,
since I was only about three seconds into the boot process. If it
happens again (and it just might, since I haven't actually managed to
create that bootable USB key for myself yet), I'll try the 'Alt-SysRq'
bit. Much like 'Ctrl-\' for those times when 'Ctrl-C' doesn't work,
'Alt-SysRq' is one of those key combos that most people know about but
never remember at the right time.

> On Wed, Nov 18, 2009 at 09:32:04PM +0530, Kapil Hari Paranjape wrote:
> > There is one more magic mantra that can be used to debug the Debian
> > (and hence Ubuntu) boot process. That is to give 'break=<value>'
> > option on the kernel command-line. (Don't forget to disable the
> > 'splash' kernel option as well!).
> >
> > Here <value> can be top, modules, premount, mount and init.
> >
> > This will introduce a 'break-point' in the boot process at the
> > designated place.
>
> Kapil, you're now our official guru for the boot process.

This is what comes out of staring too hard at the init script in the
initrd, I suppose. However, I could have saved myself the trouble
since there is documentation at 'man initramfs-tools' as I just
discovered!