Overview

The regression was actually caught and fixed on trunk on July 12th, 2010

The regression had been nominated for blocking on July 2nd, 2010, but at the time it was not called a regression and the seriousness was not conveyed (if known)

Firefox 3.6.8 was released on July 23rd, 2010

Things that went right

The bug was nominated for branch blocking, so it wasn't a case of the bug being ignored

Crash-stats quickly showed the issue/spike, making it easy to see the impact

Quick turn-around on 3.6.8

A regression that could have had similar effects was caught before shipping

Communication from affected parties was good, quick, and got to us

Sony contacted via Live-Chat, issue was recognized and escalated

Metrics saw it with a new trending URL/spidering system (before and after release)

Things that went wrong

Bug was fixed on trunk without an additional push to get it into the branches

The issue was not caught in unit testing, automated testing, QA manual testing, and living-on, even though it was essentially 100% reproducible

Is the ITMS store in the topsites? If not, should it be?

Christian actually caught it, but didn't raise alarms and wasn't able to tie it back to the regression bug. In the hand-off email to beltzner:

Firefox 3.6.7 is trending a little higher in crashes than 3.6.6 (http://crash-stats.mozilla.com/daily?form_selection=by_version&p=Firefox&v[]=3.6.7&throttle[]=100&v[]=3.6.6&throttle[]=10&v[]=&throttle[]=100&v[]=&throttle[]=100&hang_type=crash&os[]=Windows&os[]=Mac&os[]=Linux&date_start=2010-07-02&date_end=2010-07-16&submit=Generate)
I'm not sure if it's statistically significant, and I don't see any crashes on the top crasher report to be alarmed about, though I am watching "strlen | nsACString_internal::Assign(char const*, unsigned int)"....but it looks like it happened on 3.6.6 as well. Don't know why it spiked, though it looks like a QT, itunes, or ITMS page could have changed.

Suggested Improvements

Provide knobs to filter out known noise in the crash-stats reports (clegnitto)

Richer bug relations in bugzilla so that queries can be more robust and regression relationships are very clear. This is being worked on by Christian. It may not have caught this issue, though it will make sure others don't slip through (clegnitto)

Crash-stats should be viewed as part of the triage meetings (clegnitto)

Tomcat is running now top-site tests (top 25000 pages) on all Release Branches and platforms for every release to catch regressions on top sites (tomcat)