This week, we're going to demonstrate how to do some very basic debugging of both your running kernel and a loaded module using the gdb debugger running in user space. But before you get too involved here, you must review last week's column so that, by the time you return here, you understand that you should have done all of the following before going any further:

Built and installed the corresponding kernel and modules, leaving the ELF-format vmlinux image file at the top of the source tree

Rebooted to the new kernel,

Checked that the file /proc/kcore does indeed exist

Installed the gdb debugger.

Once all that's done, you can carry on. But not until then.

NOTE: This column is based heavily on the corresponding gdb debugging section from Linux Device Drivers, 3rd Edition, Chapter 4. Credit where credit is due. This is ongoing content from the Linux Foundation training program. If you want more content, please consider signing up for one of these classes. The archive of all previous "Kernel Newbie Corner" articles can be found here.

So What Exactly Are We About To Do?

What we're about to do is demonstrate an admittedly hacky way of listing the values of various kernel variables in real time, both from your running kernel and from any of your loaded modules.

Note well, though, that this is not what you'd do in an ideal situation. There are numerous better ways to do kernel debugging, most of which we'll get to in upcoming columns. But for a quick-and-dirty way of examining some of those kernel space values, as long as you have all of the prerequisites in the list above, what we're about to demonstrate will work just fine.

NOTE: While we're going to use gdb for this debugging, those familiar with gdb's normal use in user space should keep in mind that you can use it for kernel debugging only in a limited sense. What we'll be using it for is simply displaying information. In this context, you can't do anything more advanced such as assigning values, setting breakpoints, single-stepping through kernel code or the like. Quite simply, you get to look, and that's all.

So What Are My "Jiffies?"

As a trivial example of dumping something from kernel space, let's pick on the current value of the jiffies variable (or jiffies_64 on a 64-bit system), which counts the number of clock ticks since system boot. (And when we display this value, don't be alarmed if it looks totally out of whack, since it isn't actually initialized to zero at boot time.)

And now, to work, where we will need root access and the vmlinux file that corresponds to the running system:

How about another one--loops_per_jiffy, which is calculated early in the boot process and is used as the basis for the infamous "BogoMips" value?

(gdb) p loops_per_jiffy $2 = 1994923 (gdb)

And there you have it. Most developers are used to invoking gdb on a user space executable and its corresponding core file. In a sense, that's exactly what we're doing above -- invoking it on an executable (vmlinux) and its corresponding core file, which is supplied via /proc/kcore. Trivial, no?

And That's It?

Of course not. So here's what you need to remember when debugging your running kernel with gdb:

Your vmlinux file must match your running kernel, or symbols and addresses won't match and you'll get garbage during your gdb session. So after you build your new kernel, don't forget to reboot to it.

The values displayed will represent what they were at the time of invoking gdb, so if you print the contents of, say, jiffies or jiffies_64 again during that same debugging session, it will be exactly the same, over and over. To refresh the core file contents, you need to run:

(gdb) core-file /proc/kcore

every time you want updated values. (That won't make any difference to the value of loops_per_jiffy, which is calculated early and never changes until another reboot.)

Unlike a loadable module, which has access to only those kernel symbols that were explicitly EXPORTed, your gdb session has access to all kernel symbols, even those declared as "static". For example, consider the following definition from the kernel source file fs/char_dev.c:

static struct kobj_map *cdev_map;

Clearly, cdev_map has static linkage and is completely inaccessible to any loaded module. But your gdb session can see it just fine:

(gdb) p cdev_map $1 = (struct kobj_map *) 0xffff88011b878000 (gdb)

This is an amazingly handy feature.

Finally, you can examine the available symbols in kernel space by simply perusing the /proc/kallsyms kernel symbols file, as in:

But wait. There's so much more. Because here's where it gets interesting.

So What About Loadable Modules?

Once you realize that you can examine the values of symbols in the kernel symbol table from user space, it should be easy to see that you can just as conveniently examine some of the values in your loadable (and loaded) modules. After all, once you load a module, it's running in kernel space so it should be just as accessible to a gdb session as everything else.

Consider the following loadable module (in the source file gdb1.c) that we're going to load and examine (we won't bother defining any "static" symbols from here on since everything that follows will apply equally to such symbols):

What we've done above is define a number of "rday"-prefixed module variables so you can verify that you can examine all of them equally well. Let's use some of the tricks we learned last week to first examine the module's symbol table:

the variable rday_1 is in the module's BSS section and should be printable

the variables rday_2 and rday_3 are both in the module's data section and should be printable as well (exporting made absolutely no difference for what we're about to do).

Keep all of the above in mind with respect to sections since that information is going to be necessary shortly.

Loading and Debugging the Module

Assuming you built that loadable module properly, load it and let's see what happens. Immediately after it's loaded, the first thing you can do is verify that the appropriate symbols are now in the kernel symbol file /proc/kallsyms file (gdb is not running yet):

That's a good sign--we can see the variables, and they seem to be in the correct ELF sections. (Note that the difference between "d" and "D" above refers to whether those symbols are visible to loaded modules--there is no difference in their visibility to your gdb session.)

At this point, we can fire up gdb exactly the way we did last time:

# gdb vmlinux /proc/kcore

but if we try to print any of those variables, we get:

(gdb) p rday_1 No symbol "rday_1" in current context. (gdb) p rday_2 No symbol "rday_2" in current context. (gdb) p rday_3 No symbol "rday_3" in current context. (gdb)

That's because we're missing the last crucial step.

Adding a Symbol File to gdb

The problem is that the current gdb session has no idea about those module symbols because we haven't educated gdb about where they are. For that, we need to pop over to the directory /sys/module/gdb1/sections, and check where the module's various ELF sections were loaded into kernel space, and then pass that information to gdb:

Note carefully how you add a module's symbols to your current gdb session with add-symbol-file: you must add the address of the module's text segment as the argument to the command, after which you need only add the addresses of whatever extra ELF sections you care about. If there was nothing in the BSS section you wanted to print, you could have omitted that option. (Note also that the literal reference to ".../gdb1.ko" is just a short cut to whatever the full filename of the module file is which you'll have to supply.)

And when you're done poking around, just "quit" out of your gdb session, then unload the module.

And That's It?

Almost. Here are a couple more things to keep in mind when debugging with gdb:

You don't need to restrict yourself to your module's text, data and BSS sections. If there's something in, say, the ".exit.data" section that interests you, you can add that section to the symbol table as well.

As with debugging symbols in the kernel image itself, the values you're going to see for module variables are the values at the time of loading the core file /proc/kcore. If you're interested in dumping a variable that might have changed since the module was loaded, you'll have to once again reload the core file before every print command:

(gdb) core-file /proc/kcore

Designing a simple module to demonstrate that last claim is left as an exercise for the reader. HINT: Use a writable parameter.

Gaz Said:

Bazis Said:

25 Feb 14

BTW, most of the manual work with the symbol loading, hooking printk(), etc. can be automated using VisualKernel, a Visual Studio plugin for debugging Linux kernel. See this tutorial for a detailed example: http://visualkernel.com/tutorials/kgdb