Systemtap Static Probes

Summary

Systemtap allows event tracing of programs when they have static probes inserted. This allows for tracing specifics of an application on a higher level that is meaningful to the application user so they don't have to know the exact source code details for tracing what is happening. Language runtimes can benefit from this by exposing events that make sense to users of those languages/runtimes.

Detailed Description

By packaging a new version of systemtap, that enables programs that already have static dtrace probe markers in their sources and by making those packages build depend on the new systemtap-sdt-devel package and recompiling them with probe points enabled, users of those packages will be able to trace any high level events that these packages provide.

Benefit to Fedora

It will be easier for developers and users to observe what is really happening on their system on a higher (application or language) level.

Scope

Work with upstream to identify any issues with the new capabilities while we activate probes in packages.

Package new version of Systemtap (including new subpackage systemtap-sdt-devel).

Identify packages that already include static user probes (see below)

Work with package maintainer to enable them in the Fedora build spec file.

Add documentation on enabled probes and how to use them with a systemtap tapset.

dmalcolm has added another example script "pyfuntop.stp" which is a top-like view of python function calls (not yet tested in rpm context)

TODO:

Double-check generated machine code

Test with and without probes, on both architectures, with both python 2 and python 3, and with multilib installs on 64-bit

Test "pyfuntop.stp"

More documentation

Send this work upstream (we have taken an out-of-tree patch to the core adding DTrace static markers (upstream RFE 4111), reworked the patch to enable it to work with SystemTap, fixed a performance issue, added a tapset to make the markers easy to use, and written an example script that uses the resulting probe points, and ported the patch to python 3).

Address error handling within the Python 3 probe.

Ideas for additional probe points:

function calls/returns (this is what the dtrace probe has)

GIL events: instrument the raw function to claim/release of the lock, then capture the times at which it happens, then render stats

threads starting/stopping

bytecode execution metrics: e.g. trace individual bytecodes; how often does LOAD_GLOBAL get invoked

exceptions being thrown

exceptions being handled (e.g. for tracking down exactly where code is "swallowing" an error)

Notes

It seems as if several of the above were dtrace-instrumented in code that was never merged into the upstream versions of the package, but instead represented as run-time add-ons or private patches for Solaris distributions. Disappointing, but perhaps we can do better and engage the respective upstream teams. This will of course take time and panache.

At least the patches tend to be very small so we have some freedom to choose between approaches (adding STAP_PROBE/whatever hooks directly to the core upstream code; or fedora local patches; or add-on shared libraries like for php/httpd).

Another approach worth considering is adding tapsets that map process.mark() events to process.function/statement() to approximate the dtrace out-of-tree patches.

How To Test

Whether systemtap and static markers are working in general can be tested by installing systemtap, kernel-debuginfo and the systemtap-testsuite. Running sudo make installcheck in /usr/share/systemtap/testsuite

When applications get static markers enabled we should add them to a testing page listing:

Package install instructions.

Setup and sample run of the application

A reference to the probe names.

And an simple example stap invocation listing markers that can be enabled.

Question: Is there a convention/template for adding such test pages for test days?
Answer: QA/Test_Days/Create

User Experience

For packages that have static probes enabled users will be able to trace high-level events, like for example database transactions, or method tracing in virtual machines through stap.

Dependencies

A new version of systemtap with the systemtap-sdt-devel subpackage.

A new version of elfutils that provides access to the new gcc debuginfo, in particular the new cfi encodings.

Any package wishing to expose existing probes in its (upstream) sources depending on systemtap-sdt-devel and adding an --enable-dtrace or equivalent to its spec file.

Contingency Plan

Even if all the tracing will not work, packages that are converted to provide static probes will not be impacted since the probe points have (near) zero overhead, so in the worse case some packages were recompiled to enable the feature, but users will still not be able to use it.

Release Notes

Systemtap has been extended to support user space tracing, and in particular to support static (dtrace compatible) markers enabled in various programs in Fedora 12. This enables users, developers and administrators a high level overview of what is going on with their system or deep down in a specific program or subsystem.

Systemtap comes with a tutorial, a language reference manual, a tapsets reference and an examples directory under /usr/share/doc/systemtap-?.?/

TODO: Should have a list of which packages were enabled with markers when finished.