Say you want to use Kdump in your environment. You setup your machines. Then, you
crash your system for testing purposes, in order to see whether Kdump can collect a memory core successfully.
However, during the Kdump kernel startup, a panic message appears on the console. It
reads: Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0). What do you do now?

There are two variations to this message and many possible reasons to why it shows up. The solution is very
simple, but it may not appear that way if you try to debug on your own or hit the Web search for solution. You
will find a million threads, all discussing this item, most of them half-baked guesses. Like the Ubuntu
initrd issue, the solution is none of the suggested items, so please read
and learn how to handle this properly.

You may see different devices and blocks listed. It can be sdb4, sdc6 or something else. Likewise, the
unknown-block can be any sequence of numbers, like (8,3) or (0,0). Whatever the exact match, the problem is
identical.

So what gives?

Before you can solve the problem, you need to understand what it is. The reason why your Kdump kernel is
panicking is because it is trying to load with bad parameters. Whatever strings are passed to the Kdump kernel
are bad and interfere with the loading.

Normally, Kdump gets its information from two sources. One, the KDUMP_COMMANDLINE
directive, as specified in the /etc/sysconfig/kdump configuration file. By
default, the directive is left empty, which means that Kdump parses the standard kernel command line. You can
check what this is by issuing cat /proc/cmdline.

But that's not all. Kdump mangles this line to make it useful. The init script located under /etc/init.d/kdump performs some search & replace changes on the collected string so it is
good enough for the Kdump kernel.

As you can see, all kinds of items are replaced and cleaned to avoid conflicts. Now, you understand how Kdump
assembles its command line, piece by piece, taking from your stock kernel, removing lots of stuff and adding
its own. But maybe it's not removing enough?

This is your problem 1: Check whether the parsed Kdump command line is clean of all kinds of weird items that
might conflict with the loading of the kernel. Pay special attention to items like mem, memmap and others,
which hardcode memory allocations.

How can you know if memory allocations are ok?

Again, we need to refer to /proc in our running kernel and see what is currently in use. Under /proc/iomem, you will see the memory reservations, which define blocks of kernel memory that
can or cannot be allocated. While some of the items may not be applicable for the Kdump kernel, you will get a
pretty good indication if you're trying to do something that conflicts with reserved segments. Here's an
example:

After the cleanup performed in the init script, if your Kdump command line contains hard-coded items that might
conflict with some of the allocations, like for instance Video RAM area or maybe one of the PCI buses, then
your kernel might panic. You must make sure you do not override essential parts necessary for proper
functioning of the system.

You will notice I am not giving you any specific instructions, because every system is different. But the
principle remains the same and should guide you regardless what computer you're fighting against right now.

Second part, initrd

Another possibility is that your system cannot open a root device because it does not have the proper drivers
to do so. Normally, basic drivers for hardware initialization are included in the initrd image. We talked about
this in the Ubuntu initrd saga. For example, you may have special hardware that needs special drivers.

Your next task to solving the problem would be to examine what modules are listed under /etc/sysconfig/kernel and then add the missing ones. For example, if you plan on using LVM,
it seems quite necessary to add the lvm module.

After you make the changes, recreate the initrd image.

/sbin/mkinitrd

On some systems, initrd may also be named initramfs under /boot, so don't panic if you see a different naming
convention. The basic principle remains the same. Moreover, some systems may not use either, so that's a
possibility, too.

Conclusion

The VFS error boils down to two things: Kdump command line and initrd modules. Make sure your system is
configured properly. Make the necessary changes, like cleaning the kernel command line a little more and/or
adding the missing modules.

The problem will usually manifest after kernel upgrades. It may also occur in your normal kernel, but this
usually happens because of bad hardware or hardware conflicts. Your best bet is to proceed slowly and use a
checklist. First, see whether the problem can be worked around by changes to the kernel command line or the
initrd configuration. After that, move to other items, like filesystem checks, hardware components, etc.

I hope this article sheds some light on your woes. At the very least, you learned a little bit more about
systematic problem solving, Kdump command line and init script, memory allocations under /proc, how to edit the
kernel configuration, and create the initrd image. Well, party on, fellas!