Adding User Space Probing to Application (heapsort example)

Introduction

SystemTap is a powerful Linux tool that allows collection of data from both the Linux kernel and user-space applications. SystemTap includes an extensive library of predefined probes and functions for the kernel (tapsets) and a convenient scripting language to do on-the-fly data reduction. SystemTap's probing capabilities can be extended to user-space applications. There are already some applications in Fedora 12 such as java-1.6.0-openjdk and postgresql that support probing by SystemTap. This simple example uses a heapsort program to show how SystemTap support can added to nearly any user-space application and include that support in the RPM.

The user-space probes allow you investigate the operation of the program without the need to recompile the program or restart the program. It also make it very easy to create simple scripts to look at interesting characteristics of program behavior, for example how long did it take to do the average postgresql query, when did any of the Java virtual machines start doing garbage collection, and how long did java garbage collection take.

To implement the userspace application probes you will need to have a kernel that supports utrace (Fedora 10 and later kernels) and the following SystemTap rpms installed on the computer:

systemtap

systemtap-runtime

systemtap-sdt-devel

The process of adding and using the user-space application probes can be broken down into the following steps:

Modify the application's source code

Modify the RPM spec file

Use the New Probes

An extremely simple heapsort sort written in C++ is used as a starting point for this example. It reads in an arbitrary number of integers from stdin terminated by a ctrl-d, sorts the integers using a heapsort algorithm, and then outputs the sorted integers. This README describes the changes made to add the SystemTap probes to the code.

Modifying Application Source Code

Even though this example is too simple to really need the auto configuration and auto make machinery, it was provided with configure.ac and Makefile.am files to make it more closely match the typical application program. The outline for changing the source code is the following:

Configuration

The first step is to add tests to the configure.ac to enable and disable the SystemTap support. The modification to the application code should not prevent the code from compiling in environments that do not have the SystemTap user-space support. The following lines in the configure.ac file control whether SystemTap support is enabled:

When the "--enable-systemtap" is used during configuration the configure.ac needs to check to determine if the dtrace script and the sdt.h header are available. The dtrace script generates a header file and a stub object file. Within the if statement for theSystemTapconfigure there is the following additional code:

Tapset Skeleton

The tapset/heapsort.stp is a file that will make it easier for people to use the SystemTap probe points in the code. It hides some of the details about the probe point from the user. The code for the tapset will be placed in tapset/heapsort.stp. Initially, tapset/heapsort.stp can be empty. There is also a very simple Makefile.am in the tapset directory to indicate how to install and remove the tapset file. The Makefile.am in the top level directory will need to indicate that there is a subdirectory with the following lines:

SUBDIRS = tapset
DIST_SUBDIRS = $(SUBDIRS)

Probe Point Declaration

The next step is to declare the probes points in the probes.d file and the arguments that they take. The probes.d contents listed below will be processed to generate the needed include file (probes.h) and stub object file (probes.o):

What is in STAP_PROBE2() is not important. The important thing is macros are now available to instrument the application code.

Adding Probes in Source Code

For each source file with added probes the following include will be needed to provide the macros:

#include "probes.h"

The is implemented in the trace.h, a very short include file that conditionally includes the probes.h and has a TRACE macro to conditionally use the tracepoints:

#include "config.h"
#ifdef HAVE_SYSTEMTAP
// include the generated probes header and put markers in code
#include "probes.h"
#define TRACE(probe) probe
#else
// Wrap the probe to allow it to be removed when no systemtap available
#define TRACE(probe)
#endif

The macros can be placed in any place that normally executable code is placed. They will be inactive until they are used by SystemTap. The arguments can be used to relay useful state information to SystemTap. For the HEAPSORT_HEAP_PLACE the location in the heap and the value being inserted into the heap are available to SystemTap.

The raw TRACE(HEAPSORT_HEAP_PLACE()) probe can be accessed with:

process("heapsort").mark("heap_place")

The raw probes are not particularly user-friendly. The following section describes how to abstract the interface and hide those details with a tapset.

Adding a Tapset

Tapsets provide an ABI that hides the details of the probe from the user. The tapsets are typically placed in /usr/share/systemtap/tapsets. The tapsets consists of aliases and local variables for the probes. The tapset can also definite SystemTap functions and local variable that make it easier to use the probes.

The following is an example probe alias for the HEAPSORT_HEAP_PLACE() probe used in the source code:

SystemTap Flag

As with the original source code, it should be possible to build the RPMs with or without the SystemTap support enabled. The variable sdt in heapsort.spec controls whether SystemTap support is enabled. The following line indicates that the SystemTap support is enabled by default:

%{!?sdt:%define sdt 1}

The SystemTap support can be turned off with the --define on the following rpmbuild line:

rpmbuild --define "sdt 0" heapsort.spec

The variable sdt will be used in the rest of the spec file to control whether the package is built with SystemTap support.

Build Dependencies

When the package is built with SystemTap support an additional BuildRequires is needed to supply the tools to generate the probes.h header and probes.o stub files:

%if %sdt
BuildRequires: systemtap-sdt-devel
%endif

Configure

In the %build section of the heapsort.spec file the configure is extended to:

Files to Install

To make life easier for users the tapset file should be installed. The following is an addition to the %file section of the spec file:

%if %sdt
%{tapsetdir}/*.stp
%endif

Using the New Probes

In addition to the regular RPM heapsort, the heapsort-debuginfo will need to be installed. This is to allow SystemTap to determine the proper type of the arguments used in the probes. One very simple script to run is heap_tap_all.stp which probes all the probes listed in the tapset and prints out data when the each time a probe fires:

probe heapsort* {
printf("%s\n", probestr);
}

This could be run with a specific instance of heapsort and generate something like the following output: