HowTo.ProbingTheKernel History

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff will give you "everything available". I found it especially useful to say

to:

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff (or rather 32767, since /proc/sys/sunrpc/*_debug doesn't do hex) will give you "everything available". I found it especially useful to say

July 29, 2008, at 02:41 PM
by Charles Lindsey -- Initial Version complete

Changed lines 13-15 from:

The kernel contains many maps and caches most of whose contents can be inspected if only you can discover the correct incantation. In particular the various caches maintained for services accessed through RPC can be inspected through the "files" /proc/net/rpc/*/content
For example, on my own slug, 'more /proc/net/rpc/nfsd.export/content' gives me\\

to:

The kernel contains many maps and caches most of whose contents can be inspected if only you can discover the correct incantation. In particular the various caches maintained for services accessed through RPC can be inspected through the "files"

/proc/net/rpc/*/content.

For example, on my own slug, 'more /proc/net/rpc/nfsd.export/content' gives me

Added line 25:

Changed line 38 from:

@@NFSDDBG_EXPORT 0x0004

to:

NFSDDBG_EXPORT 0x0004

Changed lines 66-67 from:

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff will give you "everything available". I found it especially useful to use NFSDDBG_EXPORT when trying to discover why exportfs had not exported all the things I had asked it to.

to:

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff will give you "everything available". I found it especially useful to say

echo 4 > /proc/sys/sunrpc/nfsd_debug

(i.e. NFSDDBG_EXPORT) when trying to discover why exportfs had not exported all the things I had asked it to.

strace is an already-available module from this site. It can be used to trace (selectively) all calls upon kernel services including, in particular syscall, which will also revel to you amazing behind-the-scenes activities involving the 'procfs' file system which, for sure, were never called directly by your application.

to:

strace is an already-available module from this site. It can be used to trace (selectively) all calls upon kernel services including, in particular syscall, which will also reveal to you amazing behind-the-scenes activities involving the 'procfs' file system which, for sure, were never called directly by your application.

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff will give you "everything available". I found it especially useful to use NFSDDBG_EXPORT when trying to discover why exportfs had not exported all the things I had asked it to.

All the output produced by this debugging appears in the Syslog. If you normally connect with your slug using ssh, then it is useful to open a second ssh window with

tail -f /var/log/messages

permanently running in it, and then you will be able to follow events as they happen.

Other facilities

I am sure there must be lots more handles to cause the kernel to reveal its internal workings just waiting to be discovered, so I hope that others will add them to this Wiki as time goes by.

When operations involving the Kernel don't go as expected, and attempts to find what is happening by studying the Kernel Source Code simply get you into a mire of confusion, what tools are available to guide you through the jungle?

to:

When operations involving the Kernel don't go as expected, and attempts to find what is happening by studying the Kernel Source Code simply get you into a mire of confusion, what tools are available to guide you through the jungle?

Changed lines 24-26 from:

However, if you look closely you will also spot calls of dprintk in the kernel source, and if you do not see the corresponding message in the Syslog, you canot deduce that that part of the kernel code had not been traversed.

to:

However, if you look closely you will also spot calls of dprintk in the kernel source, and if you do not see the corresponding message in the Syslog, you cannot deduce that that part of the kernel code had not been traversed because dprintk is actually a macro (<linux/sunrpc/debug.h>) that you have to activate by writing suitable magic numbers into /proc/sys/sunrpc/*_debug. Here are the magic numbers I have identified so far:

Probing the Kernel

to:

{:title Probing the Kernel:}

Changed lines 15-18 from:

The kernel contains many maps and caches most of whose contents can be inspected if only you can discover the correct incantation. In particular the various caches maintained for services accessed through RPC can be inspected through the "files" /proc/net/rpc/*/contentFor example, on my own slug, @@

to:

The kernel contains many maps and caches most of whose contents can be inspected if only you can discover the correct incantation. In particular the various caches maintained for services accessed through RPC can be inspected through the "files" -> /proc/net/rpc/*/contentFor example, on my own slug, more /proc/net/rpc/nfsd.export/content gives me->

Syslog and printk

Most Syslog messages that you see (/var/log/messages) arise from calls of printk in the kernel. and which you can locate in the kernel source code if you grep it diligently enough.

However, if you look closely you will also spot calls of dprintk in the kernel source, and if you do not see the corresponding message in the Syslog, you canot deduce that that part of the kernel code had not been traversed.

Probing the Kernel

When operations involving the Kernel don't go as expected, and attempts to find what is happening by studying the Kernel Source Code simply get you into a mire of confusion, what tools are available to guide you through the jungle?

Kernel Debuggers

Although there are packages available such as KDB, KGDB and LTT, they all require the kernel itself to be patched before they can be used. Slugos includes no such patches (and maybe they would occupy more space in the Slug Flash than there is room for), but if someone is prepared to do the necessary groundwork, TPTB might well be prepared to incorporate them. So, in the meantime, let us see what other tools are available.

Strace

strace is an already-available module from this site. It can be used to trace (selectively) all calls upon kernel services including, in particular syscall, which will also revel to you amazing behind-the-scenes activities involving the 'procfs' file system which, for sure, were never called directly by your application.

Maps and Caches

The kernel contains many maps and caches most of whose contents can be inspected if only you can discover the correct incantation. In particular the various caches maintained for services accessed through RPC can be inspected through the "files" /proc/net/rpc/*/contentFor example, on my own slug, @@