DTrace and Erlang: a status report

I thought it’d be a good idea to write a status report about my work with Erlang, DTrace, and some SystemTap.

Quick, I Want an Overview and Some Slides….

I gave a presentation at the Erlang User Conference in Stockholm a couple weeks ago. The topic was about adding DTrace probes to the Erlang virtual machine. The presentation slides (and soon video, I hope) are available online at the Erlang Factory website. Click on the easel icon to fetch a PDF copy of the slides. When the video is available, it will probably appear as an embedded video player/applet/thingie.

I’ve given a few other talks at EUC, Erlang Factory, and Erlang Workshop venues. I’ve never had as many people tell me afterward that they both enjoyed the talk and are looking forward to using DTrace (or SystemTap) with Erlang programs.

Now we wait and see what Ericsson’s tests show. The DTrace patches were submitted well ahead of the code freeze date for the Erlang/OTP R15B01 release, which is tentatively scheduled for December 14, 2011. If all goes according to plan, the patches will be in the official Erlang/OTP R15B01 release next month!

What Is Traceable?

Take a look at the “examples” directory in the dtrace application for some simple tracing scripts. To see the actual definition of the DTrace probes and their arguments, please see the erlang_dtrace.d definition file. Please note that this link is a static one and may not always show the absolutely newest, freshest, most current version of the file.

Here’s a short summary of what’s traceable today using DTrace:

Processes: spawn, exit, hibernate, scheduled, …

Messages: send, queued, received, exit signals

Memory: GC minor & major, proc heap grow & shrink

Data copy: within heap, across heaps

Function calls: function & BIF & NIF, entry & return

Network distribution: monitor, port busy, output events

Ports: open, command, control, busy/not busy

Drivers: callback API 100% instrumented

efile_drv.c file I/O driver: 100% instrumented

As of today, there are 60 probes available.

DTrace & Erlang: Supported Platforms

I’ve been testing the following platforms regularly:

OS X 10.6.x, a.k.a. Snow Leopard. I don’t know of any reason why it shouldn’t work with 10.7.x, a.k.a. Lion.

Solaris 10. I’ve run into difficulties with long-running D scripts on my Solaris 10 machine, but I’ll write more about that in a later blog posting.

Linux, specifically CentOS 5 and CentOS 6 using SystemTap‘s DTrace compatibility API. I’ll include some recipes for setting up CentOS boxes to be capable of running user-space SystemTap in another blog posting. Things should also work when using a Fedora Core 16 system, but I haven’t tested that yet.

I’m hoping to be able to get FreeBSD 9 supported also. At the time of this writing, FreeBSD 9.0RC1 is now available for testing. Andrew Thompson @ Basho has started the FreeBSD 9 work and passed it on to me. I’ll try to get it hammered into working condition over this weekend.

How do I try it?

First, clone a copy of my source repository at GitHub:

git clone git://github.com/slfritchie/otp.git

Then you need to choose the branch that you wish to build:

Run the following if you want to compile the R14B04 version of Erlang/OTP: “cd otp ; git checkout dtrace-r14b04“.

Run the following if you want to compile the R15A version of Erlang/OTP: “cd otp ; git checkout dtrace-r15“. NOTE that this branch is not fully synchronized with Ericsson’s R15A development branch.

Run the following if you want to compile the patches as I submitted them to Ericsson for inclusion in Ericsson’s source repo: “cd otp ; git checkout dtrace-review3“.

I also have a branch called “dtrace-experiment+michal2” that contains the primary copy of all of the changes that I later merge into the three branches listed above.

Here are the instructions for building the Erlang/OTP package. I assume that you already have the toolchain dependencies (e.g. GCC compiler) and libraries (e.g. OpenSSL) already installed.

4 Responses to DTrace and Erlang: a status report

I was just wishing today that Erlang had a DTrace provider — staring at truss output is a very poor substitute for what DTrace can do when given the right provider, but right now DTrace has little insight into Erlang’s internals.