In order to get assertions down, we should make reftests fail, per-test, when they assert, after exposing a global assertion counter from XPCOM. We can annotate the manifest with the number of assertions known per test, potentially as a range:
asserts(1) // always has one assertion
asserts(5,7) // has 5-7 assertions
asserts-if(MOZ_WIDGET_TOOLKIT=="gtk2",3)
asserts-if(MOZ_WIDGET_TOOLKIT=="cocoa",0,1) // might or might not have 1 assert
Then we can report unexpected failure if the actual assertion count is too high and unexpected pass if it is too low.

Benjamin suggested nsIDebug2, which makes it easier than adding a new service.
(Also a question about the next patch: I assume I should be using getService to get the nsIDebug rather than createInstance; XPCOM already enforces it being a singleton either way.)

I think I'll need to land this disabled initially, both until we set up tinderboxes and perhaps until we do something about max-total-viewers assertions that affect the next 30 or so tests...
(The disabling would be trivial; comment out the line that gets the real assertion count, and substitute assignment of 0 instead.)
Note that this makes us fail the first test if there's an assertion during startup. I think that's a good thing.

Comment on attachment 355888[details][diff][review]
patch 2: make reftest report failures for assertions
>diff --git a/layout/tools/reftest/README.txt b/layout/tools/reftest/README.txt>+ asserts(count)
>+ Loading the test and reference is known to assert exactly
>+ count times.
>+ NOTE: asserts() with a non-zero count or max-count suppresses
>+ use of a cached canvas for the test with the annotation.
"suppresses" is correct here, of course, but I get cognitive dissonance with "asserts()" because I read it as a verb. Perhaps "An asserts() annotation with a non-zero count..."?
>+ asserts(min-count,max-count)
>+ Loading the test and reference is known to assert between
>+ min-count and max-count times.
Add a ", inclusive" here.
>+ NOTE: See above regarding canvas caching.
>+
>+ asserts-if(condition,count)
>+ asserts-if(condition,min-count,max-count)
>+ Same as above, but only if condition is true.
Why use a comma instead of a hyphen for ranges? A hyphen would be much more understandable at first glance, I think. Looks like a two-character change at first glance to do this.
>diff --git a/layout/tools/reftest/reftest.js b/layout/tools/reftest/reftest.js>+ } else if ((m = item.match(/^asserts\((\d+)(,\d+)?\)$/))) {
>+ cond = false;
>+ minAsserts = Number(m[1]);
>+ maxAsserts = (m[2] == undefined) ? minAsserts
>+ : Number(m[2].substring(1));
>+ } else if ((m = item.match(/^asserts-if\((.*?),(\d+)(,\d+)?\)$/))) {
>+ cond = false;
>+ if (Components.utils.evalInSandbox("(" + m[1] + ")", sandbox)) {
>+ minAsserts = Number(m[2]);
>+ maxAsserts =
>+ (m[3] == undefined) ? minAsserts
>+ : Number(m[3].substring(1));
>+ }
You could use the unary + operator rather than Number if you wanted here for less visual noise...
Please add a check somewhere that |minAsserts <= maxAsserts|; fail-fast is better, I think, than fail-when-actually-run.

For the record, I'm slightly uncomfortable with this attribute because it only tells you whether XPCOM was compiled with debug, not whether the app or anything else was compiled with debug. Tthey can be separate, at least in the case of XR apps... it used to be that you could compile DEBUG for only certain gecko modules, but I'm pretty sure that magic is either gone or not working. But since that's unusual, and the attribute helps solve a problem at hand, let's do it anyway.

(In reply to comment #14)
> it used to be that you could compile DEBUG for only certain gecko modules
--enable-debug-modules=xpcom,foo,bar
[you have to compile XPCOM debug if you want to compile anything else debug]
[also comes in a ^module version should you want to override --enable-debug]

And note that the above list reflects that http://hg.mozilla.org/mozilla-central/rev/4700e3c42868 has landed (so doesn't list assertions from reftest-print for Windows and Linux), except it does *not* reflect that I need to adjust that code to reflect that there are three assertions on Mac.