Although we still have ~30k (probably just 25k after the next merge) failures, we still have >300k passing. We probably want to start doing something to ensure that we don't regress what we have already passed.
I have had a detailed list about the reason of each failure. [1] We can probably land that list somehow, and mark the test green if all failures are expected.
I haven't figured out what is the best way to land this list. Potential solutions include:
1. Just do the coarse-grained thing that marks tests with any failure with "fail-if = stylo". This is easiest, but some of the mochitests include tons of different failures which we may want to detect separately.
2. Add another manifest file which records the failure patterns and counts, and pass this information to test runner, then make SimpleTest._logResult reports TEST-KNOWN-FAIL for them, and maintain a count for each pattern. Finally, log an unexpected failure in SimpleTest.finish() if there is any number mismatched.
3. Add an extra program to filter the result log, and only output patterns whose number doesn't match, and output log items which do not match any known pattern.
The third way is basically what I'm doing locally. But fitting it into the automation may take more effort than second I guess? I feel that the second approach might be the simplest?
[1] https://gist.github.com/upsuper/082268f59e6f267d5a466c2cbe698e58

I'm going to land the patches as-is in autoland. The tasks wouldn't be green immediately, since there are several dependencies haven't been landed, and there are several failures fixed in the mean time. I'll land followups to adjust the expectation later.