Linux.Conf.Au | Abstracts

Dynamic Probes - Debugging by Stealth

Dynamic Probes - Debugging by Stealth - Update and Work in Progress
Dynamic Probes has been around in the Linux Community for a couple
years. Originally it was conceived as a stealth system debugger - i.e.
one that you might be more happy to use in a production environment -
its capability and application has expanded way beyond these modest
beginnings. Since its initial release it has undergone continual
development and enhancement. During 2002 DProbes has undergone major
surgery: not only have we enhancing its capability, but we have
streamlined the kernel patch and making it more generic. In June 2002 we
released a major usability tool - the DProbes C Compiler (dpcc). DProbes
was also ported to Power and zSeries architectures in 2002. Finally but
not least DProbes was made a ready candidate for the 2.5 kernel and for
several major distributions and has in fact been accepted by TurboLinux
and MontaVista at the time of writing. So much has changed it seems an
opportune moment to report on recent developments and work in progress.
In this paper we will cover the following topics:

The essential mechanisms that allow DProbes to be a system-wide
debugger (kernel and multiple user contexts).

How global probes are implemented using breakpoint mechanisms but
without impacting the physical memory footprint through SOW. The
techniques used here are applicable to kernel debuggers in general
and would provide a means of allowing them to become system-wide
debuggers.

The new patch organisation into Kernel Probes; User Probes and
Watchpoint Probes will be discussed in detail. These patches
provide interfaces that allow any kernel module to define a probe
and probe handler. It is by this means that we have been able to
separate as an entirely optional component the original Reverse
Polish Notation (RPN) interpreter that was previously an integral
part of DProbes.

How new pagepoint extension: we have the option of exploiting the
processor's paging mechanism to extend numbers of concurrent
watchpoints beyond the few that are permitted by processor
debugging h/w.

The new dpcc compiler: this allows probepoints to be defined using
c-like language that references the variables and structures
present in the code which is being probed. The Probe definition
appears as an insert to the original source, without the need to
edit or recompile the original source.

We will also spend some time discussing how DProbes is able to be
used as a general debugging engine to trigger crash dumps and core
dumps from arbitrary code locations; to provide an on-the-fly
tracing and logging mechanism (dynamic trace), which has found
particular favour with some kernel developers whose needs was to
add temporary printks for debugging; and finally we will show how
DProbes can be used to exercise error code paths through fault
injection.