Meta

W3C’s testharness comes to WebKit

With the help of WebKit contributors, we recently landed a patch that made the W3C JavaScript test harness available for use by WebKit test authors – providing a new set of opportunities for reuse and portability between the W3C and WebKit test repositories.

In the old world, before the landing of this patch, both the WebKit and the W3C test repositories had unique JavaScript frameworks used by test authors to make common test assertions (for example, checking the return value of a function against an expected value). The frameworks were similar in functionality, yet had unique APIs that made tests only compatible with the test suite they were targeted for, and made the effort to convert a test from using one framework to the other potentially very time consuming.

A patch for WebKit Bug 80709 landed the W3C test harness (testharness.js and testharnessreport.js) in the WebKit repository. We had to make a small change to the default output of testharness.js – specifically – stripping the HTML formatted table and instead outputting the test results in plain text. This change was required for compatibility with the WebKit test infrastructure, and was implemented in testharnessreport.js.

The fact that we now have the W3C JavaScript test harness in the WebKit repository means that W3C JavaScript tests can be copied into the WebKit repository with only a single change – an update to the path of the two test harness scripts. This also means that WebKit test authors now have the choice of using js-test-pre.js (the WebKit-specific JavaScript test framework) or testharness.js to make assertions in their JavaScript tests. Going forward, testharness.js is preferable, even for tests targeted only for the WebKit repository, as it allows for portability between the two test repositories in the future.

In order to use the W3C test harness in a JavaScript test all the test author needs to do is include the two scripts in the main test file. For example:

<script src=”../../resources/testharness.js”></script>

<script src=”../../resources/testharnessreport.js”></script>

The relative path above would need to be updated based on the exact location of the test file in question. The scripts are located in the WebKit repository under the LayoutTests/resources directory. Instructions for how to use testharness.js are detailed in the comment section of the script file itself.

Building off of this new capability, an effort is underway to define a process for importing entire W3C test suites into the WebKit test repository. The ability to run W3C-style ref tests already existed, and now that JavaScript tests can be run as well, no technical barriers remain to running all automated W3C tests through WebKit test bots.

A process for importing third party tests was drafted at the April 2012 WebKit Contributors Meeting and captured on the WebKit wiki here. This process is currently being discussed and refined by the community. Once it has been finalized, W3C test suites will be identified and imported into the WebKit repository.

This new capability allows not only for the reuse of W3C tests by the WebKit community, it also lets test authors write a test once for both W3C and WebKit, decreasing the effort required to write tests while increasing the test coverage for each repository and improving the ability to detect regressions.

On June 15th-16th, Adobe will be hosting an event called Test the Web Forward in San Francisco with the goal of writing and contributing tests to the W3C repository. The test-a-thon will focus on tests for a number of specs: CSS3 Background & Borders, CSS3 Transforms, CSS Object Model (CSSOM), and potentially also CSS3 Multi-Column (still under discussion). Participants will be free to choose to write either JavaScript or Ref Tests, and now that both will work with the WebKit repository, all of these tests will hopefully make their way into WebKit through the import process mentioned above. Registration is open so if anyone is interested in learning more about W3C tests, visit the Test the Web Forward site for more information.

Maintaining and expanding a robust set of tests is essential to ensuring that both existing and new web features provide the best experience possible to Internet users. If you are passionate about the web, please join us for the test-a-thon and help Test the Web Forward!