Wednesday, February 18, 2009

We get a lot of bug reports on the JRuby project. No, it's not because JRuby's a buggy POS. It's because we're implementing the core of an entire platform. It's hard.

Over the past year or so we kinda let the bug tracker go a little wild. We've never had more than two full-time people on the project, and community contributions wax and wane with the seasons, so we've had to choose our battles. For some amount of time, we would focus on performance, since without a reasonably-performing implementation, nobody would choose to use JRuby. Then we'd cycle back to fixing bugs, usually in the last couple weeks before a release. So the flow of bug fixes waxed and waned with our release cycles.

The problem with this approach was that we really couldn't keep up with all the reports coming in and continue to work on the "unreported issues" of JRuby like performance. So something had to give, and it was sometimes the bugs.

Roughly around September, we crossed an important threshold: 500 open bugs. They dated as far back as 2006 (we moved to Codehaus in 2006), and across the entire release sequence of JRuby; there were even some bug reports against pre-1.0 versions. From September until about mid-January, we worked to keep the number of open bugs below 500. Sometimes it would peek above, but we were generally successful.

About mid-January, I figured it was time to really bite the bullet, and I started walking from the oldest bug forward. It has been a long slog, but I'm making great progress.

So far, I've managed to whittle the total number of bugs down from 506 to 326, and that's with a couple dozen more filed in the past two weeks. The majority of these were either already fixed or invalid/incomplete bugs, but I've also made a concentrated effort to focus on and fix as many as possible. At the very least, I've made an attempt on every single one.

During that process, I learned a bit about how my brain works. During the past year, if I didn't know immediately how to fix a bug I often walked away from it and worked on something fun like compiler optimization. But during this latest bug hunt, because I did not want to just punt all those bugs one more time, I did something a little different.

Whenever I encountered a bug that seemed difficult or that I simply didn't want to fix, I would do one of two things:

Take a break. Often this meant I had reached my limit and needed a breather. A little Wii, a glass of scotch, a movie...anything to take my mind of it for a bit.

Count to ten. Sometimes, after reading through the bug, all I really needed was a moment for my brain to start "wondering" about the solution. Once the wheels started turning, I was almost compelled to dig deeper.

As a result, many long-standing bugs--including bugs I thought we would never fix--are now resolved and ready for the 1.2 release. I'm going on three weeks now of 12-hour days trying to fix what seems like an endless stream of bugs, but the end is in sight. And I want to stress that this level of effort has never happened in a single release. There's as much work in JRuby 1.2 as in multiple 1.1.x releases combined. I'll be blogging more of these posts over the next couple weeks.

For your amusement, here is the complete list of bugs we have resolved during the past nine weeks, since shortly after the 1.1.6 release. These aren't all fixes, but even the "already fixed", "won't fix", "not a bug", or "incomplete" ones were reviewed, tested, and considered. A couple of these I just applied patches or marked resolved based on someone else's work. And of course we're not done with JRuby 1.2 yet. Just you wait and see :)

Thanks for a great thing you are doing guys!Maybe posting some documentation about how some things are done inside the JRuby interpreter would make us less techie guys more able to contribute to the project. Or some "what you can do" trails.

Thanks for a great thing you are doing guys!Maybe posting some documentation about how some things are done inside the JRuby interpreter would make us less techie guys more able to contribute to the project. Or some "what you can do" trails.

Thanks all for the words of encouragement. I think we'll have more bugs fixed in 1.2 than the last couple releases combined. And that's on top of performance and feature enhancements.

Mirko: That's a great suggestion, I'll have to look into it.

DevDanke: We have had a lot of community help; if we hadn't, the bug tracker could have been well over 500. But a lot of the bugs were in obscure parts of JRuby only one or two of us know well, so they ended up sitting for a long time.

And yes, I'm going to take a break after 1.2, that's for sure :)

Anonymous: Yeah we need more architecture documents. You might try poking around the JRuby Wiki for docs there, and hopefully we'll get some better ones up some day soon. There's also my talk on InfoQ about JRuby internals...it might help give a general idea how things fit together.

rene.scheibe: I just had Jira export to an Excel document, opened that in OpenOffice, and copy/pasted it into blogger.

Yeah, seriously, I second DevDanke. Don't burn out with the 12 hour days. Is there anything those of us with full-time jobs can do to help -- sending you guys pizza or scotch money to help you take the edge off?