DTrace is many things to many people. To me it is a tool for engaging with complexity. Sure there’s an important place for the DTrace Toolkit, advanced OpenStorage analytics, Chime and other wonderful technologies built on DTrace (most of which don’t even come close to exposing the user to the more low-level cranium challenging detail), but for me DTrace remains “The One True Tool” (as slashdot reviewer) and the means by which I can ask an arbitrary question and get an instant answer.

When presenting DTrace to a new audience, I see my primary goal as creating desire. Nothing worth having comes easily. Getting to grips with DTrace involves a steep learning curve. Before exposing candidates to potentially overwhelming detail, I need to show them why the gain is going to be worth the pain. It’s also useful to underline some seeds of self doubt and insecurity, to establish my authority as the teacher they can trust. So I generally start by talking about complexity.

All I’m going to blog here is one of my favourite complexity stories. It is best done live, with lots of stuff scrolling up a green screen, and plenty of theatrical flare. However, for the purpose of this post I’ve done the UNIX thing and used a pipe into the wc(1) command. I’m sorry if it loses something in the telling, but the base data is still interesting.

I usually start by talking about how complexity has increased during my time at Sun. In the good old days when we all programmed in C it was possible for one person to have a handle on the whole system. But today’s world is very different. In a bid to connect with the old timers, we start talking about “Hello World!”. I then show how good the truss(1) utility is at exposing some of the implementation detail.

We then move on to a Java implementation. The code looks similar, and it is functionally equivalent. Although both the C and Java versions complete in far less then a second, even the casual observer can see that the Java variant is slower. I then start digging deeper with truss(1). First, we compare just the number of system calls, then the number of inter-library function calls, the lastly, the number of intra-library function calls.

This post is really just the raw data, simply to underline the point that todays software environments are a lot more complex than we often give them credit for; and secondly, that we need a new generation of tools to engage with this level of complexity. For added fun, I’ve added Perl and Python data to the mix. Enjoy!

Context

Conclusion

Of course the above gives no indication of how long each experiment took. Yes, I could have wrapped the experiment with ptime(1), but I'll leave that as an exercise for the reader. When I use this illustration with a live audience, it's generally sufficient to allow the longest case to continue to scroll up the screen for the rest of the presentation.

At this point, I generally move on. Usually, I say some kind words about high level languages, abstraction, code reuse etc. I am not out to knock Java. That's not the point. The point is complexity. I then move on to how DTrace can help us to engage with complexity. I'd do that here, but I hope that I'll continue to be asked to speak on the subject, and I don't want to give it all away here, just now.

What I learned by taking a Dtrace class was that Dtrace does NOT explain the internals of Solaris… it just makes them observable.

To use Dtrace well one needs to spent the time learning Solaris internal details.

To use Dtrace casually you just need to follow the one-liners and higher level tools created by the folks that do know Solaris… like Mike, Brian and Adam (M-B-A) that created Storage Analytics (a few years after they created dtrace).

I wish every major OS had something very similar to dtrace… some do have something baguely similar but a truly instrumented kernel and user land code is required. Not a small effort.

I often use DTrace to help me understand the inner workings of Solaris. For example, one can learn a huge amount about code flow with just a few nested FBT traces. Add to that the fact that you can browse the source online, and then throw in the McDougall/Mauro internals book for good measure, and a lot of understanding can be gained in very little time.

The bulk of my DTrace use is just a handful or so of one-liners.
A lot of my D scripts are less than 100 lines long, and they generally evolve as I explore (just a small change here or there to gain the next nuance of understanding). The most important ingredient is an inquisitive mind.