Systemtap is a tool that allows developers and administrators to write
and reuse simple scripts to deeply examine the activities of a live
Linux system. Data may be extracted, filtered, and summarized quickly
and safely, to enable diagnoses of complex performance or functional
problems.

NOTE: This tutorial does not describe every feature available in
systemtap. Please see the individual stap manual pages for
the most up-to-date information. These may be available installed on
your system, or at http://sourceware.org/systemtap/man/.

The essential idea behind a systemtap script is to name events,
and to give them handlers. Whenever a specified event occurs,
the Linux kernel runs the handler as if it were a quick subroutine,
then resumes. There are several kind of events, such as entering or
exiting a function, a timer expiring, or the entire systemtap session
starting or stopping. A handler is a series of script language
statements that specify the work to be done whenever the event occurs.
This work normally includes extracting data from the event context,
storing them into internal variables, or printing results.

Systemtap works by translating the script to C, running the system C
compiler to create a kernel module from that. When the module is
loaded, it activates all the probed events by hooking into the kernel.
Then, as events occur on any processor, the compiled handlers run.
Eventually, the session stops, the hooks are disconnected, and the
module removed. This entire process is driven from a single
command-line program, stap.

Figure:
A systemtap smoke test.

This paper assumes that you have installed systemtap and its
prerequisite kernel development tools and debugging data, so that you
can run the scripts such as the simple one in
Figure . Log on as root, or even better,
login as a user that is a member of stapdev group or as a
user authorized to sudo, before running systemtap.

Figure:
A taste of systemtap: a system-wide strace, just for
the open system call.