Firefox 33 Post mortem

Our aurora/beta populations is not representative for graphics cards/drivers

In general we need a better way to communicate with our aurora/beta users. We have enough users, we just need to get better quality data

Some people might have thought "It is so big that someone is going to report it"

Having old machines with no up to date drivers to test during the cycle

(lmandel) More broadly, we need to come up with a way to get better information about potential graphics related issues. This may involve setting up a graphics lab, contracting another company that already has one, measuring beta and trying to improve the graphics coverage on this channel,...

Need about:support data submitted with feedback and support posts.+1

about:telemetry as well

We have been blind most of the process.

We couldn't turn OMTC off because of regressions and lack of testing

We should add a best practice here about testing features in on/off states

Need to do more testing on XP

Email process needs to be cleaned up.

(mschifer) lurking crashes like h264 and other noise we haven't seen these crashes

Can we expand the scope of what we look at

(bsmedberg) concerned about case where the browser is broken and we don't crash

need to be able to collect data

(bsmedberg) very poor signal of users just stopping to use beta

talked to metrics about getting some data (UA has ideas, can we get in on this meeting)

Turning off OMTC for one beta release killed our ability to see problems

(mschifer) spread out big feature landings in the same component

(cww) when we turn on/off a feature for one release it's hard to tell what the data is afterwards

heard about black screen issues after beta 6 but assumed it was related to disabling omtc

(cww) don't get feedback for startup crashes or anything that makes the browser unusable

(cww) very difficult to get about:support and other information when Firefox can't start

(cww) detect when people are installing on top of an existing installation

start in safe mode?

ask why they're reinstalling

(cww) can we tell if people are downgrading? bug 1091881

(cww) we can do outreach in advance if we expect something to break (OMTC-sized) but we didn't realize it was going to have the possible impact it would.

Blacklist update broken

Nobody seems to understands it

We don't have tests

We should rewrite it

(lmandel) At the very least, there are serious limitations even if it's working and we don't see to have good ways to test changes to the list. bjacob has a bug on file and we should pursue a project the build a replacement. https://bugzilla.mozilla.org/show_bug.cgi?id=838845

We should be able to know which users a blacklist change will affect (in advance), based on their FHR profile data

the pkix tests should be launched during the beta cycle

Last minute patch with certificate - bug 1058812

No way to properly test with OMTC = off.

For every new features, TBPL should run with pref = on & pref = off during the aurora & beta cycle

we mixed two important changes at the same time:

OMTC = on

D3D11

I didn't even realize this until very late (Cww)

Try on mozilla-release has some failures "because it is mozilla-release and tests are not planned for that"

--> Not much that can be done about this in the short term given how Try is configured.

(tracy) pulling the plug when we know we're shipping broken product

33.1 handcuffed us

would it have been better to delay release a couple of days and push a build to beta (hindsight)

QA doesn't have a good collection of machines to test things on

mschifer working on that

Regressed bug 933733 (how?)

bug 937306 was supposed to fix this, QA called for help in verification, apparently didn't communicate well enough to UA about this.