Layout Test Expectations and Baselines

The primary function of the LayoutTests is as a regression test suite; this means that, while we care about whether a page is being rendered correctly, we care more about whether the page is being rendered the way we expect it to. In other words, we look more for changes in behavior than we do for correctness.

All layout tests have “expected results”, or “baselines”, which may be one of several forms. The test may produce one or more of:

A text file containing JavaScript log messages.

A text rendering of the Render Tree.

A screen capture of the rendered page as a PNG file.

WAV files of the audio output, for WebAudio tests.

For any of these types of tests, baselines are checked into the LayoutTests directory. The filename of a baseline is the same as that of the corresponding test, but the extension is replaced with -expected.{txt,png,wav} (depending on the type of test output). Baselines usually live alongside tests, with the exception when baselines vary by platforms; read Layout Test Baseline Fallback for more details.

Lastly, we also support the concept of “reference tests”, which check that two pages are rendered identically (pixel-by-pixel). As long as the two tests' output match, the tests pass. For more on reference tests, see Writing ref tests.

Failing tests

When the output doesn't match, there are two potential reasons for it:

The port is performing “correctly”, but the output simply won't match the generic version. The usual reason for this is for things like form controls, which are rendered differently on each platform.

The port is performing “incorrectly” (i.e., the test is failing).

In both cases, the convention is to check in a new baseline (aka rebaseline), even though that file may be codifying errors. This helps us maintain test coverage for all the other things the test is testing while we resolve the bug.

If a test can be rebaselined, it should always be rebaselined instead of adding lines to TestExpectations.

Bugs at crbug.com should track fixing incorrect behavior, not lines in TestExpectations. If a test is never supposed to pass (e.g. it‘s testing Windows-specific behavior, so can’t ever pass on Linux/Mac), move it to the NeverFixTests file. That gets it out of the way of the rest of the project.

There are some cases where you can‘t rebaseline and, unfortunately, we don’t have a better solution than either:

Reverting the patch that caused the failure, or

Adding a line to TestExpectations and fixing the bug later.

In this case, reverting the patch is strongly preferred.

These are the cases where you can't rebaseline:

The test is a reference test.

The test gives different output in release and debug; in this case, generate a baseline with the release build, and mark the debug build as expected to fail.

The test is flaky, crashes or times out.

The test is for a feature that hasn't yet shipped on some platforms yet, but will shortly.

Handling flaky tests

The flakiness dashboard is a tool for understanding a test’s behavior over time. Originally designed for managing flaky tests, the dashboard shows a timeline view of the test’s behavior over time. The tool may be overwhelming at first, but the documentation should help. Once you decide that a test is truly flaky, you can suppress it using the TestExpectations file, as described below.

We do not generally expect Chromium sheriffs to spend time trying to address flakiness, though.

How to rebaseline

Since baselines themselves are often platform-specific, updating baselines in general requires fetching new test results after running the test on multiple platforms.

Rebaselining using try jobs

The recommended way to rebaseline for a currently-in-progress CL is to use results from try jobs, by using the command-tool third_party/WebKit/Tools/Scripts/webkit-patch rebaseline-cl:

First, upload a CL.

Trigger try jobs by running webkit-patch rebaseline-cl. This should trigger jobs on tryserver.blink.

Wait for all try jobs to finish.

Run webkit-patch rebaseline-cl again to fetch new baselines. By default, this will download new baselines for any failing tests in the try jobs. (Run webkit-patch rebaseline-cl --help for more specific options.)

Commit the new baselines and upload a new patch.

This way, the new baselines can be reviewed along with the changes, which helps the reviewer verify that the new baselines are correct. It also means that there is no period of time when the layout test results are ignored.

Options

Rebaselining with try jobs

The tests which webkit-patch rebaseline-cl tries to download new baselines for depends on its arguments.

By default, it tries to download all baselines for tests that failed in the try jobs.

If you pass --only-changed-tests, then only tests modified in the CL will be considered.

You can also explicitly pass a list of test names, and then just those tests will be rebaselined.

If some of the try jobs failed to run, and you wish to continue rebaselining assuming that there are no platform-specific results for those platforms, you can add the flag --fill-missing.

Rebaselining manually

If the tests is already listed in TestExpectations as flaky, mark the test NeedsManualRebaseline and comment out the flaky line so that your patch can land without turning the tree red. If the test is not in TestExpectations, you can add a [ Rebaseline ] line to TestExpectations.

Run third_party/WebKit/Tools/Scripts/webkit-patch rebaseline-expectations

Post the patch created in step 2 for review.

Kinds of expectations files

TestExpectations: The main test failure suppression file. In theory, this should be used for temporarily marking tests as flaky.

NeverFixTests: Tests that we never intend to fix (e.g. a test for Windows-specific behavior will never be fixed on Linux/Mac). Tests that will never pass on any platform should just be deleted, though.

SlowTests: Tests that take longer than the usual timeout to run. Slow tests are given 5x the usual timeout.

StaleTestExpectations: Platform-specific lines that have been in TestExpectations for many months. They‘re moved here to get them out of the way of people doing rebaselines since they’re clearly not getting fixed anytime soon.

Flag-specific expectations files

Updating the expectations files

Ordering

The file is not ordered. If you put new changes somewhere in the middle of the file, this will reduce the chance of merge conflicts when landing your patch.

Syntax

The syntax of the file is roughly one expectation per line. An expectation can apply to either a directory of tests, or a specific tests. Lines prefixed with # are treated as comments, and blank lines are allowed as well.

The syntax of a line is roughly:

[ bugs ] [ "[" modifiers "]" ] test_name [ "[" expectations "]" ]

Tokens are separated by whitespace.

The brackets delimiting the modifiers and expectations from the bugs and the test_name are not optional; however the modifiers component is optional. In other words, if you want to specify modifiers or expectations, you must enclose them in brackets.

Lines are expected to have one or more bug identifiers, and the linter will complain about lines missing them. Bug identifiers are of the form crbug.com/12345, code.google.com/p/v8/issues/detail?id=12345 or Bug(username).

If no modifiers are specified, the test applies to all of the configurations applicable to that file.

Expectations can be one or more of Crash, Failure, Pass, Rebaseline, Slow, Skip, Timeout, WontFix, Missing, NeedsManualRebaseline. If multiple expectations are listed, the test is considered “flaky” and any of those results will be considered as expected.

For example:

crbug.com/12345 [ Win Debug ] fast/html/keygen.html [ Crash ]

which indicates that the “fast/html/keygen.html” test file is expected to crash when run in the Debug configuration on Windows, and the tracking bug for this crash is bug #12345 in the Chromium issue tracker. Note that the test will still be run, so that we can notice if it doesn't actually crash.

Assuming you're running a debug build on Mac 10.9, the following lines are all equivalent (in terms of whether the test is performed and its expected outcome):