A new way of tracking leak reports

We get lots of leak reports from users. There is a spectrum of quality.

Some are hopelessly vague and will never lead to anything useful. (“After browsing for several hours, Firefox is using 100s of MBs of memory. This is unacceptable; please fix.”) Bug 643177 is an example.

Some are very precise. This makes them easy to reproduce, likely to be fixed quickly, and easy to re-confirm if other leaks are fixed in the interim. (“I managed to reduce the problem down to the attached 10 line HTML file, it causes my machine to run out of memory within 10 seconds of loading.”) Bug 654106 is a good example.

Most are somewhere between these two extremes.

Because many of the reports aren’t great, it can be hard to tell if the problem is still present some time later. A single leak may be reported N times, then fixed, and N-1 reports stay open. In short, leak reports get stale. (This is true of many bug reports, but I think leak reports are more prone to staleness than most.)

How bugs are currently tracked

There is a keyword, ‘mlk’, which is added to almost all leak reports. There are over 600 open bugs with that keyword, going back over 10 years. So it’s not much use.

In the lead-up to Firefox 4, I used bug 632234 (which I’ll henceforth call “mlk-fx4-old”) to track potentially blocking leaks. It worked well.

After that, I created bug 640452 (which I’ll henceforth call “mlk-fx5+”), with which I’ve been tracking leaks in the lead-up to Firefox 5 and later versions. I carried over unresolved bugs from mlk-fx4-old. mlk-fx5+ is starting to fill up feel stale. Basically, I can see it suffering the same problems as the ‘mlk’ keyword before too long.

So I’m thinking about changing how these are tracked. The basic idea is to use keep using the ‘mlk’ keyword for all leak reports, and then have one leak-tracking bug for each version of Firefox, so it’s clear which version each report applies to.

Steps needed to start this

Add the ‘mlk’ keyword to all mlk-fx4-old and mlk-fx5+ bugs that lack it.

Open new tracking bugs: mlk-fx4-beta, mlk-fx4, mlk-fx5, mlk-fx6. (The Firefox 4 beta period was long enough, and there were enough leak reports filed against beta versions, that separating mlk-fx4-beta and mlk-fx4 seems worthwhile.) Make each mlk-fxN depend on mlk-fx(N-1).

For all the existing bugs tracked by mlk-fx4-old and mlk-fx5+, add them to the appropriate new tracking bug. With one exception: for hopelessly vague ones, just mark them as duplicates of mlk-fxN, with an explanatory message (“we’re not ignoring leaks, look at all these ones we’re tracking! but your report doesn’t tell us anything we don’t already know, sorry”).

Close mlk-fx5+.

Steps needed to maintain this in the future

When Firefox version N’s cycle starts, open mlk-fxN, and mark it as depending on mlk-fx(N-1).

For all new leak reports, mark it as blocking mlk-fxN, for appropriate N. Also add the ‘mlk’ keyword.

If someone confirms in a comment that a problem reported in version N is still present in version N+1, mark that bug as also blocking mlk-fx(N+1).

Properties of this system

You can still search for all leak reports, based on the ‘mlk’ keyword.

You can immediately tell roughly how stale a report is likely to be, based on which mlk-fxN tracking bug it blocks. This is more reliable than the bug number or file date; for example, we are still getting reports against Firefox 4 even though Firefox 5 (which has fixed a number of leaks) is in beta and Firefox 6 just went to Aurora. This immediately gives a starting priority for all leak reports: more recent ones have higher priority because they’re more likely to still be unfixed.

Tracking bugs shouldn’t get too big and unwieldy, because each Firefox version has a limited lifespan.

Reports against version N still block mlk-fx(N+1), but via one level of indirection. Reports against version N+2 still block mlk-fx(N+2), but via two levels of indirection, etc. So the full chain of dependencies is maintained.

We could periodically go through older bugs (eg. 3 releases ago) and ask people to re-confirm, and close out ones that get no response. But we wouldn’t have to do that.

Am I crazy?

Is this bureaucratic overkill? I don’t think so. It’ll take some work, but I’m happy to do that. It’ll only take an hour or two to set up, and then it won’t be much harder to maintain than what I’m currently doing with the mlk-fx5+ bug. (I also have plans for writing instructions to help users file better leak reports.) And it’ll allow us to proceed much more usefully with the lists of leak reports that we have.

But I’m interested to hear if you disagree, or have any ideas for improving it. Thanks!

Brian: possibly after some time (eg. 3 versions) we could say “we’ll close this if you can’t reconfirm it on the latest Firefox”. Or not; Bugzilla is full of unclosed bugs so more isn’t necessarily a problem, esp. since the per-version tracking means that still-open bugs against old versions don’t really clutter up things for newer versions.