From the WPT doc, reftest-wait is a mechanism similar to testRunner's waitUntilDone / notifyDone():
"Controlling When Comparison Occurs
By default reftest screenshots are taken after the load event has fired, and web fonts (if any) are loaded. In some cases it is necessary to delay the screenshot later than this, for example because some DOM manipulation is required to set up the desired test conditions. To enable this, the test may have a class="reftest-wait" attribute specified on the root element. This will cause the screenshot to be delayed until the load event has fired and the reftest-wait class has been removed from the root element. Note that in neither case is exact timing of the screenshot guaranteed: it is only guaranteed to be after those events."
The reftest-wait attributed is already used by the following tests imported into WebKit:
./FileAPI/url/url_xmlhttprequest_img.html
./css/css-grid/abspos/absolute-positioning-changing-containing-block-001.html
./css/css-grid/abspos/grid-item-absolute-positioning-dynamic-001.html
./css/css-grid/abspos/grid-positioned-item-dynamic-change-001.html
./css/css-ui/text-overflow-005.html
./css/css-ui/text-overflow-017.html
./css/css-ui/text-overflow-021.html
./css/mediaqueries/min-width-tables-001.html
./css/selectors/focus-within-001.html
./css/selectors/focus-within-002.html
./css/selectors/focus-within-003.html
./css/selectors/focus-within-004.html
./css/selectors/focus-within-005.html
./css/selectors/focus-within-006-expected.html
./css/selectors/focus-within-006.html
./css/selectors/focus-within-007.html
./css/selectors/focus-within-008.html
./css/selectors/focus-within-010.html
./css/selectors/focus-within-shadow-001.html
./css/selectors/focus-within-shadow-002.html
./css/selectors/focus-within-shadow-003.html
./css/selectors/focus-within-shadow-004.html
./css/selectors/focus-within-shadow-005.html
./css/selectors/focus-within-shadow-006.html
./css/selectors/selector-placeholder-shown-type-change-001.html
./css/selectors/selector-placeholder-shown-type-change-002.html
./css/selectors/selector-placeholder-shown-type-change-003.html
./css/selectors/selector-read-write-type-change-002.html
./css/selectors/selector-required-type-change-002.html
./mathml/relations/html5-tree/href-click-1.html
./mathml/relations/html5-tree/href-click-2.html
Probably, most of them pass because the expected final rendering happens fast enough. However, this is a bit fragile and might lead to flacky tests. Attached is a patch that moves WPT's reftest_wait_0 test so that one can run it with Tools/Scripts/run-webkit-tests reftest_wait_0.html. As indicated in the test description, it is expected to fail since:
- reftest_wait_0.html is a doc with a red background but after 2 seconds the background color is changed to green (The end of the test is controlled via the reftest-wait attribute).
- reftest_wait_0-expected.html is a doc with a red background.
However, because webkit-test-runner does not support reftest-wait, the screenshot of reftest_wait_0.html is actually taken before the background color is turned into green.

Created attachment 341562[details]
Patch (proof of concept)
This patch shows how to emulate reftest-wait with some helper JS script that relies on testRunner's waitUntiDone/notifyDone. I wonder if we should make the test runner script inject such JS code or if class="reftest-wait" should be handled in any other way...

Created attachment 342130[details]
Archive of layout-test-results from ews103 for mac-sierra
The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103 Port: mac-sierra Platform: Mac OS X 10.12.6

Created attachment 342134[details]
Archive of layout-test-results from ews104 for mac-sierra-wk2
The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104 Port: mac-sierra-wk2 Platform: Mac OS X 10.12.6

Created attachment 342135[details]
Archive of layout-test-results from ews114 for mac-sierra
The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews114 Port: mac-sierra Platform: Mac OS X 10.12.6

Created attachment 342137[details]
Archive of layout-test-results from ews201 for win-future
The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews201 Port: win-future Platform: CYGWIN_NT-6.1-2.9.0-0.318-5-3-x86_64-64bit

Created attachment 342140[details]
Archive of layout-test-results from ews121 for ios-simulator-wk2
The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews121 Port: ios-simulator-wk2 Platform: Mac OS X 10.13.4

Created attachment 342142[details]
Archive of layout-test-results from ews101 for mac-sierra
The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews101 Port: mac-sierra Platform: Mac OS X 10.12.6

Created attachment 342143[details]
Archive of layout-test-results from ews105 for mac-sierra-wk2
The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews105 Port: mac-sierra-wk2 Platform: Mac OS X 10.12.6

Created attachment 342144[details]
Archive of layout-test-results from ews121 for ios-simulator-wk2
The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews121 Port: ios-simulator-wk2 Platform: Mac OS X 10.13.4

Created attachment 342145[details]
Archive of layout-test-results from ews200 for win-future
The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews200 Port: win-future Platform: CYGWIN_NT-6.1-2.9.0-0.318-5-3-x86_64-64bit

Created attachment 342146[details]
Archive of layout-test-results from ews114 for mac-sierra
The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews114 Port: mac-sierra Platform: Mac OS X 10.12.6

Do you know how Chrome is handling ref-test?
If we have to handle various classes of documents, maybe it would be better to handle them in JS directly.
Like updating testharnessreport.js to handle these classes, or maybe a script dedicated to handle these classes.
This would require updating the tests to include that script (which would be specialized for each browser) so we would need to discuss this on WPT GitHub first.

(In reply to youenn fablet from comment #30)
> Do you know how Chrome is handling ref-test?
>
No, I have not checked. I would need to investigate that...
> If we have to handle various classes of documents, maybe it would be better
> to handle them in JS directly.
> Like updating testharnessreport.js to handle these classes, or maybe a
> script dedicated to handle these classes.
> This would require updating the tests to include that script (which would be
> specialized for each browser) so we would need to discuss this on WPT GitHub
> first.
Yeah, I had initially tried testharnessreport.js, however reftests generally don't include such scripts. Injecting the script in the TestRunner is maybe not the best approach but I'm not sure it will be better to have the import script insert a script or to have to modify all the WPT reftests that use reftest-wait...

Created attachment 342193[details]
Archive of layout-test-results from ews206 for win-future
The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews206 Port: win-future Platform: CYGWIN_NT-6.1-2.9.0-0.318-5-3-x86_64-64bit

(In reply to Ms2ger from comment #35)
> We're definitely not requiring upstream that reftests must contain a script,
> so something along the lines of what's proposed here makes sense to me.
Why not? If the way the tests identify the time to record the actual result is removing the class name from the document element, why can't that code simply invoke a function instead? Given the amount of boilerplate WPT requires elsewhere, that seems to have a low cost.
I'm not at all excited to hard-code this insane ad-hoc convention WPT has into WTR or DRT.

Comment on attachment 342158[details]
Patch
View in context: https://bugs.webkit.org/attachment.cgi?id=342158&action=review>>> Tools/Scripts/webkitpy/port/driver.py:510
>>> + command += "'--check-reftest-wait"
>>
>> Something seems to be off about the quotes at the start here.
>
> The single quote is just a separator between arguments, it does not need to be closed (see the rest of the _command_from_driver_input function for more context).
Why do we need to have a runtime option for this? Why can't we always check for this condition.
> Tools/WebKitTestRunner/InjectedBundle/InjectedBundle.cpp:404
> +static const char* checkReftestWaitScript =
I don't think it makes sense to implement a logic like this in JS if we're modifying WTR
since we already have a code to wait for sub-resources to load.
If we're going with the injected script route, we're much better off adding a generic mechanism to inject arbitrary JS into the test.
I really don't want WPT specific logic creeping into WTR or DRT. r- because of this.

Comment on attachment 342158[details]
Patch
View in context: https://bugs.webkit.org/attachment.cgi?id=342158&action=review>>>> Tools/Scripts/webkitpy/port/driver.py:510
>>>> + command += "'--check-reftest-wait"
>>>
>>> Something seems to be off about the quotes at the start here.
>>
>> The single quote is just a separator between arguments, it does not need to be closed (see the rest of the _command_from_driver_input function for more context).
>
> Why do we need to have a runtime option for this? Why can't we always check for this condition.
Not sure I understand your question, but this --check-reftest-wait command is passed to the WTR and DRT binaries. I'm not sure they know whether we are executing a reftest or whether a test is a WPT, unless you suggest to copy the logic into WTR and DRT too.
>> Tools/WebKitTestRunner/InjectedBundle/InjectedBundle.cpp:404
>> +static const char* checkReftestWaitScript =
>
> I don't think it makes sense to implement a logic like this in JS if we're modifying WTR
> since we already have a code to wait for sub-resources to load.
> If we're going with the injected script route, we're much better off adding a generic mechanism to inject arbitrary JS into the test.
> I really don't want WPT specific logic creeping into WTR or DRT. r- because of this.
I also didn't like this approach, it was mostly a first attempt to open discussion however I was surprised that Chromium people actually did the same thing and supported this approach in a mail exchanges Youenn and I had with them. One argument in favor of this is that the sync logic is nontrivial, apparently one needs to properly check that web fonts are ready before taking the screenshot. This is needed even for tests without reftest-wait class, so that is one more argument in favor of a generic injection mechanism (another argument is that the same code would probably need to be injected in DRT too).

(In reply to Ryosuke Niwa from comment #38)
> Ugh... this is so ugly. Why can't WPT define a specific JS function helper
> for browsers to hook in instead of relying on having some browser specific
> detection of the class name?
(In reply to Ryosuke Niwa from comment #39)
> (In reply to Ms2ger from comment #35)
> > We're definitely not requiring upstream that reftests must contain a script,
> > so something along the lines of what's proposed here makes sense to me.
>
> Why not? If the way the tests identify the time to record the actual result
> is removing the class name from the document element, why can't that code
> simply invoke a function instead? Given the amount of boilerplate WPT
> requires elsewhere, that seems to have a low cost.
Regarding this reftest-wait class, I see tree options:
1) Implement support in WebKit
2) Implement a conversion mechanism in the script importer/exporter (reftest-wait <-> wait/done)
3) Convince upstream WPT to implement a new wait/done mechanism (and update tests)
(2) seems a bit fragile. We already have import/export issue with reftests, so I don't really want to make that worse.
reftest-wait is already a well-established convention inherited from initial 2007 implementation in Mozilla and is used everywhere in upstream WPT, so I doubt (3) will be easy, but we can try.
(1) is what I tried here and that's what Chromium people decided too and they seem to be happy with that.
We've already discussed this by private emails with Youenn and some Google/WPT people. Maybe we should open a public discussion with the rest of the browser vendors and WPT authors...