Thursday, January 31, 2008

When I read the posting about Cylon, the Ruby debugger for Visual Studio by SapphireSteel, I immediately thought about the Rubinius debugger highlighted in InfoQ recently. I decided to give it a try and see what I came up with. I wasn’t able to replicate Dermot’s tests exactly (I don’t know where he set his breakpoints, and Rubinius doesn’t support everything he tested), but here are my initial results:

Line Based

Rubinius no debugger

Rubinius w/debugger no breakpoints

Ruby

2.039 seconds

2.399 seconds

0.898 seconds

Call Based

Rubinius no debugger

Rubinius w/debugger no breakpoints

Ruby

13.441 second

13.999 seconds

4.171 seconds

Rubinius still has a long performance road ahead of them in terms of general execution, but it’s pretty exciting to see that their debugger is so fast. The cylon debugger (which smoked the other options in their testing) was 25% slower for line based debugging, and 306% slower for call based. By comparison, Rubinius’ debugger only added 17% for line based and 4% for call based (actually, I’m guessing that had the line based run through more iterations, the Rubinius debugger would have done better, I think most of its time is spent in set up).

Using the Rubinius debugger requires that you add a call to the debugger to the program:

I've seen a number of comparisons regarding the speeds of various debuggers, usually in regard to touting some new debugger as the fastest.

Although most of the comparisons I've read are well-intentioned and may give some representative idea, the real story however is a little bit more complicated.

There are several kinds of ways a debugger can get called into action initially, and get used subsequently; the fact that a number of the comparisons gloss over this suggests that those doing the comparisons might not even be aware of what's going on. For example consider this statement from the cited article:

The full speed debugger gives Rubinius a big edge versus Ruby implementations that need to rely on tracing based debuggers (no matter how fast these implementations are).

I'll come back to this comment a little later.

In the early days of debugging, one invoked the debugger at the outset. Perhaps one set up some breakpoints or started stepping through the program. It is for that kind of behavior that the debugger speed comparisons are most relevant.

Nowadays however it is not uncommon to bring in the program sometime after the program has started executing. In Ruby and Python (and even Bash ;-) you can "require", "import", or "source" the debugger anywhere in the code. And when this is done there is no overhead, until the debugger is pulled in. In fact, it's possible to set up an interrupt handler so that when the program gets a signal, that's when the debugger is loaded. (I can't remember, but I think I did this in a Python debugger I worked on.)

Debugging in Ruby is a little bit weird, different, good, and bad. (And in 1.9 I suspect it's going to get weirder, more different, better and worse). In contrast to say Perl, Ruby has less debugging support built in, and no Ruby primitives for accessing stack frames. Part of what made the default Ruby debugger "debug.rb" particularly slow is creating and saving frame information along with a "binding" to access variables. I think it has been noted in ruby-core and perhaps by Evan Phoenix that saving bindings can be particular slow.

In ruby-debug, Kent Sibilev's solution to debugging is particularly interesting and unique. There are three steps one must go through. First you bring in the debugger (e.g. require "ruby-debug"). Second you start recording frame information (e.g. Debugger.start), and third you add a trace hook (e.g. Debugger.debugger) which goes into the debugger proper. The interesting step is the second step, somewhat necessitated by Ruby's lack of built-in support. So not only is there a Debug.start, but there is also a Debugger.stop which removes the trace hook.

Therefore a savvy programmer who runs into debugger speed performance issues can speed up debugging by a judicious use of start/stop. In practice people like to put ruby-debug's start somewhere towards the beginning of execution, but it need not be done that way. If there's a place that you want go into the debugger at, you could but the start right before the call to debugger. In fact I think a call to debugger without a start will run it for you. The downside of this is just that you won't get call stack information lower than where the start was issued. However in terms of performance, you in fact have zero any debugger overhead before the first start and after any subsequent stop. (Sort of, it's possible to nest starts and stops; so I really mean stopping out of the the first-level.)

Ruby is not unique here. In Python debuggers (e.g. pdb and pydb) which a trace hook as well, there is a little hack that has been added to reduce overhead: if there are no line breakpoints set and one is not "stepping", the line event handler is not called. Python has the ability to set a break on a return frame call to a routine.

But having written all this, I admit that most people don't really know what's going on in the various debuggers so timing tests may be helpful here. In Rails 2.0 debug support was added; but it is done by issuing a Debugger.start early on. My take is that it was deemed that the performance degradation of ruby-debug not significant that it caused people to give it a second thought.

Don't get me wrong. I really like a VM trap instruction or instruction replacement (which I believe goes back to the days of debugging assembler on the IBM System/370 because in fact there wasn't a TRAP instruction like the PDP/11 had). And I think instruction replacement on the VM will have a big and bright future on Ruby as well.

I'm just saying the trace-hook versus trap thing might not necessarily be black and white as folks make it out to be.

I wrote the above after a long tiring day at work. There are a couple of small errors I'd like to correct. Debugger.start is what installs a trace hook and that's when there starts to be overhead until Debugger.stop (level 1) is executed; "debugger" calls the debugger read loop.

Although "import"ing a Python debugger had been discussed in pydb, the code to do this isn't part of the code base; however there is a little used (and I'm not even sure if it works) "server" mode in pydb which does install a trap handler to pull in the debugger. However the bash debugger does allow one to set up a signal handler to bring in the debugger.

Finally, to elaborate on why debugging in 1.9 may get weirder is the fact that there may be mixed levels of debugging: there may be instruction replacement of VM code as well as trace hook debugging