For make benefit of glorious tubes

Top Crashers By Url and MTBF

Working with ss and chofman, we’ve created 2 new types of reports: a Top Crashers by Url and a Mean Time Before Failure (MTBF).

Given the current state of performance of the non-report parts of Socorro’s webapp, most of the thought and time have gone into the backend piece of these reports. You can read about the ReportDatabaseDesign on the project’s wiki.

Top Crashers by URL

On which websites do our browser builds crash the most? Which curses do our users hurl at us when this happens?

This report uses the optional url feild of a crash report to answer this question. It has two modes byurl and bydomain. You can read more about the details on TopCrashersByUrl. Crashes which have a comment, include the comment and a link to the actual crash report. Don’t worry, personal details have been removed, we don’t tie a specific user to a specific url.

We will be putting links into Socorro to these new reports, with the work neilio is doing, but for now here are various links.

These reports are for a release in general as well as stats for Mac and Win, allowing for drilling down into OS. Several frontend enhancements to this report are coming.

, Product and versions in these reports include:

Firefox 3.0.4

Firefox 3.0.5

Firefox 3.1a2

Firefox 3.1b1

Firefox 3.1b2

Firefox 3.0.4pre

Firefox 3.0.5pre

Firefox 3.0.6pre

Firefox 3.1b3pre

Firefox 3.1b2pre

Thunderbird 3.0a3

Thunderbird 3.0b1

Thunderbird 3.0b1pre

Thunderbird 3.0b2pre

I’ve gotten a good dose of feedback on tweaks to make and bugs to fix, but hopefully you’ll find these new reports useful. Tomcat has already mentioned augmenting his list of urls to populate his test automation for 3.1 (using spider to test most popular urls) with the urls in these reports.

5 responses

It would also be interesting to have a “top crashers by domain/url” that’s weighted by the site’s popularity. In a perfect (Big Brother) world this would be “crashes per visit”, but using something like comScore rankings might be a good proxy.

This would allow looking for sites that are inherently crashy for us. In other words, the top 3 domains on the list (facebook, google, youtube) are there in large part just because they have so much traffic, not because Firefox is terribly crashy there. Well, probably. There’s a lot of JS on Gmail, so who knows? Maybe we are abnormally crashy, it’s hard to say either way with the current data.

What’s with the spikes on some of the MTBF graphs? Surely we’re not putting out random builds that are amazingly stable? Are these data glitches that can be filtered out, so the graphs don’t look like flat lines with a few spikes?

A MTBF list that’s binned MTBF counts (10 users crashed after 30 minutes, 20 users after 60 minutes, etc) would also be interesting to see. I wouldn’t expect crashes to vary that much by day-of-release — it’s the same code.

It would also be interesting to see if the crash reporter could include the number of page loads for the session (or something similar), to look at stable-but-idle vs stable-and-busy.

Forgive the rant, but someone at Mozilla needs to start putting real labels on their graph axises. Log or semi log plots are nice too. Either that, or scale the axis to get rid of the noise points which mean… well… nothing.

when drilling down on OS Firefox 3.0.5 Mac shows 15E6 seconds of MTBF on the third day, however there are only 259 200 seconds elapsed in this time. This is probably just a script error like dividing by 0.

The frontend of MTBF got the least love in this release as most of the time was on the backend. Justin, we will be smoothing values over 10 day average, as well as refining start and stop dates for the window.

You could use stats like Alexa Rank, Google PageRank (page-based not site-based) or (US-only) Compete.com as well.

Compete.com data you can download a dump file (with license restrictions but I imagine they’d give it free for the free publicity they’d get). There are backdoor ways of getting the other two stats, and probably other useful stats out there too.