NAME

ddb -- interactive kernel debugger

SYNOPSIS

In order to enable kernel debugging facilities include:
optionsKDBoptionsDDB
To prevent activation of the debugger on kernel panic(9):
optionsKDB_UNATTENDED
In order to print a stack trace of the current thread on the console for
a panic:
optionsKDB_TRACE
To print the numerical value of symbols in addition to the symbolic
representation, define:
optionsDDB_NUMSYM
To enable the gdb(1) backend, so that remote debugging with kgdb(1) is
possible, include:
optionsGDB

DESCRIPTION

The ddb kernel debugger is an interactive debugger with a syntax inspired
by gdb(1). If linked into the running kernel, it can be invoked locally
with the 'debug' keymap(5) action. The debugger is also invoked on
kernel panic(9) if the debug.debugger_on_panicsysctl(8) MIB variable is
set non-zero, which is the default unless the KDB_UNATTENDED option is
specified.
The current location is called dot. The dot is displayed with a
hexadecimal format at a prompt. The commands examine and write update
dot to the address of the last line examined or the last location
modified, and set next to the address of the next location to be examined
or changed. Other commands do not change dot, and set next to be the
same as dot.
The general command syntax is: command[/modifier] address[,count]
A blank line repeats the previous command from the address next with
count 1 and no modifiers. Specifying address sets dot to the address.
Omitting address uses dot. A missing count is taken to be 1 for printing
commands or infinity for stack traces.
The ddb debugger has a pager feature (like the more(1) command) for the
output. If an output line exceeds the number set in the lines variable,
it displays ``--More--'' and waits for a response. The valid responses
for it are:
SPC one more page
RET one more line
q abort the current command, and return to the command input mode
Finally, ddb provides a small (currently 10 items) command history, and
offers simple emacs-style command line editing capabilities. In addition
to the emacs control keys, the usual ANSI arrow keys may be used to
browse through the history buffer, and move the cursor within the current
line.

COMMANDS

examinex Display the addressed locations according to the formats in the
modifier. Multiple modifier formats display multiple locations.
If no format is specified, the last format specified for this
command is used.
The format characters are:
b look at by bytes (8 bits)
h look at by half words (16 bits)
l look at by long words (32 bits)
a print the location being displayed
A print the location with a line number if possible
x display in unsigned hex
z display in signed hex
o display in unsigned octal
d display in signed decimal
u display in unsigned decimal
r display in current radix, signed
c display low 8 bits as a character. Non-printing
characters are displayed as an octal escape code (e.g.,
'\000').
s display the null-terminated string at the location. Non-
printing characters are displayed as octal escapes.
m display in unsigned hex with character dump at the end of
each line. The location is also displayed in hex at the
beginning of each line.
i display as an instruction
I display as an instruction with possible alternate formats
depending on the machine:
amd64 No alternate format.
i386 No alternate format.
ia64 No alternate format.
powerpc No alternate format.
sparc64 No alternate format.
S display a symbol name for the pointer stored at the
address
xf Examine forward: execute an examine command with the last
specified parameters to it except that the next address displayed
by it is used as the start address.
xb Examine backward: execute an examine command with the last
specified parameters to it except that the last start address
subtracted by the size displayed by it is used as the start
address.
print[/acdoruxz]
p[/acdoruxz]
Print addrs according to the modifier character (as described
above for examine). Valid formats are: a, x, z, o, d, u, r, and
c. If no modifier is specified, the last one specified to it is
used. The argument addr can be a string, in which case it is
printed as it is. For example:
print/x "eax = " $eax "\necx = " $ecx "\n"
will print like:
eax = xxxxxx
ecx = yyyyyy
write[/bhl] addrexpr1 [expr2...]
w[/bhl] addrexpr1 [expr2...]
Write the expressions specified after addr on the command line at
succeeding locations starting with addr. The write unit size can
be specified in the modifier with a letter b (byte), h (half
word) or l (long word) respectively. If omitted, long word is
assumed.
Warning: since there is no delimiter between expressions, strange
things may happen. It is best to enclose each expression in
parentheses.
set $variable [=] expr
Set the named variable or register with the value of expr. Valid
variable names are described below.
break[/u]
b[/u] Set a break point at addr. If count is supplied, continues count
- 1 times before stopping at the break point. If the break point
is set, a break point number is printed with '#'. This number
can be used in deleting the break point or adding conditions to
it.
If the u modifier is specified, this command sets a break point
in user address space. Without the u option, the address is
considered to be in the kernel space, and a wrong space address
is rejected with an error message. This modifier can be used
only if it is supported by machine dependent routines.
Warning: If a user text is shadowed by a normal user space
debugger, user space break points may not work correctly.
Setting a break point at the low-level code paths may also cause
strange behavior.
deleteaddrdaddrdelete #numberd #number
Delete the break point. The target break point can be specified
by a break point number with '#', or by using the same addr
specified in the original break command.
watchaddr,size
Set a watchpoint for a region. Execution stops when an attempt
to modify the region occurs. The size argument defaults to 4.
If you specify a wrong space address, the request is rejected
with an error message.
Warning: Attempts to watch wired kernel memory may cause
unrecoverable error in some systems such as i386. Watchpoints on
user addresses work best.
hwatchaddr,size
Set a hardware watchpoint for a region if supported by the
architecture. Execution stops when an attempt to modify the
region occurs. The size argument defaults to 4.
Warning: The hardware debug facilities do not have a concept of
separate address spaces like the watch command does. Use hwatch
for setting watchpoints on kernel address locations only, and
avoid its use on user mode address spaces.
dhwatchaddr,size
Delete specified hardware watchpoint.
step[/p]
s[/p] Single step count times (the comma is a mandatory part of the
syntax). If the p modifier is specified, print each instruction
at each step. Otherwise, only print the last instruction.
Warning: depending on machine type, it may not be possible to
single-step through some low-level code paths or user space code.
On machines with software-emulated single-stepping (e.g., pmax),
stepping through code executed by interrupt handlers will
probably do the wrong thing.
continue[/c]
c[/c] Continue execution until a breakpoint or watchpoint. If the c
modifier is specified, count instructions while executing. Some
machines (e.g., pmax) also count loads and stores.
Warning: when counting, the debugger is really silently single-
stepping. This means that single-stepping on low-level code may
cause strange behavior.
until[/p]
Stop at the next call or return instruction. If the p modifier
is specified, print the call nesting depth and the cumulative
instruction count at each call or return. Otherwise, only print
when the matching return is hit.
next[/p]
match[/p]
Stop at the matching return instruction. If the p modifier is
specified, print the call nesting depth and the cumulative
instruction count at each call or return. Otherwise, only print
when the matching return is hit.
trace[/u] [pid | tid] [,count]
t[/u] [pid | tid] [,count]
where[/u] [pid | tid] [,count]
bt[/u] [pid | tid] [,count]
Stack trace. The u option traces user space; if omitted, trace
only traces kernel space. The optional argument count is the
number of frames to be traced. If count is omitted, all frames
are printed.
Warning: User space stack trace is valid only if the machine
dependent code supports it.
search[/bhl] addrvalue [mask] [,count]
Search memory for value. This command might fail in interesting
ways if it does not find the searched-for value. This is because
ddb does not always recover from touching bad memory. The
optional count argument limits the search.
showallprocs[/m]
ps[/m] Display all process information. The process information may not
be shown if it is not supported in the machine, or the bottom of
the stack of the target process is not in the main memory at that
time. The m modifier will alter the display to show VM map
addresses for the process and not show other information.
showallttys
Show all TTY's within the system. Output is similar to pstat(8),
but also includes the address of the TTY structure.
showallchains
Show the same information like "show lockchain" does, but for
every thread in the system.
showalllocks
Show all locks that are currently held. This command is only
available if witness(4) is included in the kernel.
showallpcpu
The same as "show pcpu", but for every CPU present in the system.
showallrman
Show information related with resource management, including
interrupt request lines, DMA request lines, I/O ports and I/O
memory addresses.
showapic
Dump data about APIC IDT vector mappings.
showbreaks
Show breakpoints set with the "break" command.
showbuffer
Show buffer structure of structbuf type. Such a structure is
used within the FreeBSD kernel for the I/O subsystem
implementation. For an exact interpretation of the output,
please see the sys/buf.h header file.
showcbstat
Show brief information about the TTY subsystem.
showcdev
Without argument, show the list of all created cdev's, consisting
of devfs node name and struct cdev address. When address of cdev
is supplied, show some internal devfs state of the cdev.
showconifhk
Lists hooks currently waiting for completion in
run_interrupt_driven_config_hooks().
showcpusets
Print numbered root and assigned CPU affinity sets. See
cpuset(2) for more details.
showcyrixreg
Show registers specific to the Cyrix processor.
showdomainaddr
Print protocol domain structure structdomain at address addr.
See the sys/domain.h header file for more details on the exact
meaning of the structure fields.
showffs [addr]
Show brief information about ffs mount at the address addr, if
argument is given. Otherwise, provides the summary about each
ffs mount.
showfileaddr
Show information about the file structure structfile present at
address addr.
showfiles
Show information about every file structure in the system.
showfreepages
Show the number of physical pages in each of the free lists.
showgeom [addr]
If the addr argument is not given, displays the entire GEOM
topology. If addr is given, displays details about the given
GEOM object (class, geom, provider or consumer).
showidt
Show IDT layout. The first column specifies the IDT vector. The
second one is the name of the interrupt/trap handler. Those
functions are machine dependent.
showinodedeps [addr]
Show brief information about each inodedep structure. If addr is
given, only inodedeps belonging to the fs located at the supplied
address are shown.
showinpcbaddr
Show information on IP Control Block structin_pcb present at
addr.
showintr
Dump information about interrupt handlers.
showintrcnt
Dump the interrupt statistics.
showirqs
Show interrupt lines and their respective kernel threads.
showjails
Show the list of jail(8) instances. In addition to what jls(8)
shows, also list kernel internal details.
showlapic
Show information from the local APIC registers for this CPU.
showlockaddr
Show lock structure. The output format is as follows:
class:
Class of the lock. Possible types include mutex(9),
rmlock(9), rwlock(9), sx(9).
name: Name of the lock.
flags:
Flags passed to the lock initialization function. For
exact possibilities see manual pages of possible lock
types.
state:
Current state of a lock. As well as flags it's lock-
specific.
owner:
Lock owner.
showlockchainaddr
Show all threads a particular thread at address addr is waiting
on based on non-sleepable and non-spin locks.
showlockedbufs
Show the same information as "show buf", but for every locked
structbuf object.
showlockedvnods
List all locked vnodes in the system.
showlocks
Prints all locks that are currently acquired. This command is
only available if witness(4) is included in the kernel.
showlocktreeshowmalloc
Prints malloc(9) memory allocator statistics. The output format
is as follows:
Type Specifies a type of memory. It is the same as a
description string used while defining the given
memory type with MALLOC_DECLARE(9).
InUse Number of memory allocations of the given type,
for which free(9) has not been called yet.
MemUse Total memory consumed by the given allocation
type.
Requests Number of memory allocation requests for the
given memory type.
The same information can be gathered in userspace with ``vmstat-m''.
showmap[/f] addr
Prints the VM map at addr. If the f modifier is specified the
complete map is printed.
showmsgbuf
Print the system's message buffer. It is the same output as in
the ``dmesg'' case. It is useful if you got a kernel panic,
attached a serial cable to the machine and want to get the boot
messages from before the system hang.
showmount
Displays short info about all currently mounted file systems.
showmountaddr
Displays details about the given mount point.
showobject[/f] addr
Prints the VM object at addr. If the f option is specified the
complete object is printed.
showpage
Show statistics on VM pages.
showpageq
Show statistics on VM page queues.
showpciregs
Print PCI bus registers. The same information can be gathered in
userspace by running ``pciconf-lv''.
showpcpu
Print current processor state. The output format is as follows:
cpuid Processor identifier.
curthread Thread pointer, process identifier and
the name of the process.
curpcb Control block pointer.
fpcurthread FPU thread pointer.
idlethread Idle thread pointer.
APICID CPU identifier coming from APIC.
currentldt LDT pointer.
spinlocksheld Names of spin locks held.
showpgrpdump
Dump process groups present within the system.
showproc [addr]
If no [addr] is specified, print information about the current
process. Otherwise, show information about the process at
address addr.
showprocvm
Show process virtual memory layout.
showprotoswaddr
Print protocol switch structure structprotosw at address addr.
showregisters[/u]
Display the register set. If the u modifier is specified, it
displays user registers instead of kernel registers or the
currently saved one.
Warning: The support of the u modifier depends on the machine.
If not supported, incorrect information will be displayed.
showrmanaddr
Show resource manager object structrman at address addr.
Addresses of particular pointers can be gathered with "show
allrman" command.
showrtc
Show real time clock value. Useful for long debugging sessions.
showsleepchain
Show all the threads a particular thread is waiting on based on
sleepable locks.
showsleepqshowsleepqueue
Both commands provide the same functionality. They show
sleepqueue structsleepqueue structure. Sleepqueues are used
within the FreeBSD kernel to implement sleepable synchronization
primitives (thread holding a lock might sleep or be context
switched), which at the time of writing are: condvar(9), sx(9)
and standard msleep(9) interface.
showsockbufaddrshowsocketaddr
Those commands print structsockbuf and structsocket objects
placed at addr. Output consists of all values present in
structures mentioned. For exact interpretation and more details,
visit sys/socket.h header file.
showsysregs
Show system registers (e.g., cr0-4 on i386.) Not present on some
platforms.
showtcpcbaddr
Print TCP control block structtcpcb lying at address addr. For
exact interpretation of output, visit netinet/tcp.h header file.
showthread [addr]
If no addr is specified, show detailed information about current
thread. Otherwise, information about thread at addr is printed.
showthreads
Show all threads within the system. Output format is as follows:
Firstcolumn Thread identifier (TID)
Secondcolumn Thread structure address
Thirdcolumn Backtrace.
showttyaddr
Display the contents of a TTY structure in a readable form.
showturnstileaddr
Show turnstile structturnstile structure at address addr.
Turnstiles are structures used within the FreeBSD kernel to
implement synchronization primitives which, while holding a
specific type of lock, cannot sleep or context switch to another
thread. Currently, those are: mutex(9), rwlock(9), rmlock(9).
showuma
Show UMA allocator statistics. Output consists five columns:
Zone Name of the UMA zone. The same string that was
passed to uma_zcreate(9) as a first argument.
Size Size of a given memory object (slab).
Used Number of slabs being currently used.
Free Number of free slabs within the UMA zone.
Requests Number of allocations requests to the given zone.
The very same information might be gathered in the userspace with
the help of ``vmstat-z''
showunpcbaddr
Shows UNIX domain socket private control block structunpcb
present at the address addrshowvmochk
Prints, whether the internal VM objects are in a map somewhere
and none have zero ref counts.
showvmopag
This is supposed to show physical addresses consumed by a VM
object. Currently, it is not possible to use this command when
witness(4) is compiled in the kernel.
showvnode [addr]
Prints vnode structvnode structure lying at [addr]. For the
exact interpretation of the output, look at the sys/vnode.h
header file.
showvnodebufsaddr
Shows clean/dirty buffer lists of the vnode located at addr.
showwatches
Displays all watchpoints. Shows watchpoints set with "watch"
command.
showwitness
Shows information about lock acquisition coming from the
witness(4) subsystem.
gdb Toggles between remote GDB and DDB mode. In remote GDB mode,
another machine is required that runs gdb(1) using the remote
debug feature, with a connection to the serial console port on
the target machine. Currently only available on the i386
architecture.
halt Halt the system.
killsigpid
Send signal sig to process pid. The signal is acted on upon
returning from the debugger. This command can be used to kill a
process causing resource contention in the case of a hung system.
See signal(3) for a list of signals. Note that the arguments are
reversed relative to kill(2).
reboot [seconds]
reset [seconds]
Hard reset the system. If the optional argument seconds is
given, the debugger will wait for this long, at most a week,
before rebooting.
help Print a short summary of the available commands and command
abbreviations.
captureoncaptureoffcaptureresetcapturestatusddb supports a basic output capture facility, which can be used
to retrieve the results of debugging commands from userpsace
using sysctl(2). captureon enables output capture; captureoff
disables capture. capturereset will clear the capture buffer
and disable capture. capturestatus will report current buffer
use, buffer size, and disposition of output capture.
Userspace processes may inspect and manage ddb capture state
using sysctl(8):
debug.ddb.capture.bufsize may be used to query or set the current
capture buffer size.
debug.ddb.capture.maxbufsize may be used to query the compile-
time limit on the capture buffer size.
debug.ddb.capture.bytes may be used to query the number of bytes
of output currently in the capture buffer.
debug.ddb.capture.data returns the contents of the buffer as a
string to an appropriately privileged process.
This facility is particularly useful in concert with the
scripting and textdump(4) facilities, allowing scripted debugging
output to be captured and committed to disk as part of a textdump
for later analysis. The contents of the capture buffer may also
be inspected in a kernel core dump using kgdb(1).
runscriptscriptsunscript
Run, define, list, and delete scripts. See the SCRIPTING section
for more information on the scripting facility.
textdumpsettextdumpstatustextdumpunset
The textdumpset command may be used to force the next kernel
core dump to be a textdump rather than a traditional memory dump
or minidump. textdumpstatus reports whether a textdump has been
scheduled. textdumpunset cancels a request to perform a
textdump as the next kernel core dump. More information may be
found in textdump(4).

VARIABLES

The debugger accesses registers and variables as $name. Register names
are as in the ``showregisters'' command. Some variables are suffixed
with numbers, and may have some modifier following a colon immediately
after the variable name. For example, register variables can have a u
modifier to indicate user register (e.g., ``$eax:u'').
Built-in variables currently supported are:
radix Input and output radix.
maxoff Addresses are printed as ``symbol+offset'' unless offset is
greater than maxoff.
maxwidth The width of the displayed line.
lines The number of lines. It is used by the built-in pager.
tabstops Tab stop width.
workxx Work variable; xx can take values from 0 to 31.

EXPRESSIONS

Most expression operators in C are supported except '~', '^', and unary
'&'. Special rules in ddb are:
Identifiers The name of a symbol is translated to the value of the
symbol, which is the address of the corresponding object.
'.' and ':' can be used in the identifier. If supported by
an object format dependent routine, [filename:]func:lineno,
[filename:]variable, and [filename:]lineno can be accepted
as a symbol.
Numbers Radix is determined by the first two letters: '0x': hex,
'0o': octal, '0t': decimal; otherwise, follow current radix.
. dot
+ next
.. address of the start of the last line examined. Unlike dot
or next, this is only changed by examine or write command.
' last address explicitly specified.
$variable Translated to the value of the specified variable. It may
be followed by a ':' and modifiers as described above.
a#b A binary operator which rounds up the left hand side to the
next multiple of right hand side.
*expr Indirection. It may be followed by a ':' and modifiers as
described above.

SCRIPTING

ddb supports a basic scripting facility to allow automating tasks or
responses to specific events. Each script consists of a list of DDB
commands to be executed sequentially, and is assigned a unique name.
Certain script names have special meaning, and will be automatically run
on various ddb events if scripts by those names have been defined.
The script command may be used to define a script by name. Scripts
consist of a series of ddb commands separated with the ; character. For
example:
script kdb.enter.panic=bt; show pcpu
script lockinfo=show alllocks; show lockedvnods
The scripts command lists currently defined scripts.
The run command execute a script by name. For example:
run lockinfo
The unscript command may be used to delete a script by name. For
example:
unscript kdb.enter.panic
These functions may also be performed from userspace using the ddb(8)
command.
Certain scripts are run automatically, if defined, for specific ddb
events. The follow scripts are run when various events occur:
kdb.enter.acpi The kernel debugger was entered as a result of an
acpi(4) event.
kdb.enter.bootflags The kernel debugger was entered at boot as a result
of the debugger boot flag being set.
kdb.enter.break The kernel debugger was entered as a result of a
serial or console break.
kdb.enter.cam The kernel debugger was entered as a result of a
CAM(4) event.
kdb.enter.mac The kernel debugger was entered as a result of an
assertion failure in the mac_test(4) module of the
TrustedBSD MAC Framework.
kdb.enter.ndis The kernel debugger was entered as a result of an
ndis(4) breakpoint event.
kdb.enter.netgraph The kernel debugger was entered as a result of a
netgraph(4) event.
kdb.enter.panic panic(9) was called.
kdb.enter.powerfail The kernel debugger was entered as a result of a
powerfail NMI on the sparc64 platform.
kdb.enter.powerpc The kernel debugger was entered as a result of an
unimplemented interrupt type on the powerpc
platform.
kdb.enter.sysctl The kernel debugger was entered as a result of the
debug.kdb.enter sysctl being set.
kdb.enter.trapsig The kernel debugger was entered as a result of a
trapsig event on the sparc64 or sun4v platform.
kdb.enter.unionfs The kernel debugger was entered as a result of an
assertion failure in the union file system.
kdb.enter.unknown The kernel debugger was entered, but no reason has
been set.
kdb.enter.vfslock The kernel debugger was entered as a result of a VFS
lock violation.
kdb.enter.watchdog The kernel debugger was entered as a result of a
watchdog firing.
kdb.enter.witness The kernel debugger was entered as a result of a
witness(4) violation.
In the event that none of these scripts is found, ddb will attempt to
execute a default script:
kdb.enter.default The kernel debugger was entered, but a script
exactly matching the reason for entering was not
defined. This can be used as a catch-all to handle
cases not specifically of interest; for example,
kdb.enter.witness might be defined to have special
handling, and kdb.enter.default might be defined to
simply panic and reboot.

HINTS

On machines with an ISA expansion bus, a simple NMI generation card can
be constructed by connecting a push button between the A01 and B01
(CHCHK# and GND) card fingers. Momentarily shorting these two fingers
together may cause the bridge chipset to generate an NMI, which causes
the kernel to pass control to ddb. Some bridge chipsets do not generate
a NMI on CHCHK#, so your mileage may vary. The NMI allows one to break
into the debugger on a wedged machine to diagnose problems. Other bus'
bridge chipsets may be able to generate NMI using bus specific methods.

SEEALSO

HISTORY

The ddb debugger was developed for Mach, and ported to 386BSD 0.1. This
manual page translated from man(7) macros by Garrett Wollman.
Robert N. M. Watson added support for ddb output capture, textdump(4) and
scripting in FreeBSD 7.1.