Design Notes for a Test Review System

Expected Workflow

Workflow for Test Author

Author designs and creates tests and, when possible, checks that the test behaves as expected in at least one implementation.

Author submits tests

Author gets back automated report of what's wrong

Author fixes tests to match format

Author resubmits tests, they pass

Author waits for review

Author receives review comments on 6/9 tests

Author fixes tests

Author resubmits tests

Author receives acknowledgement that tests have been accepted and checked in

Workflow for Reviewer

If the Reviewer has 'Owner' or 'Peer' status (see review), the Reviewer searches the submittal data base for tests in the 'Accepted' state; if not, or if no 'Accepted' tests were found, Reviewer searches the submittal data base for tests in the 'Resubmitted' or 'Submitted' state and selects a test to review. (He or she cannot review his or her own tests.)

Reviewer looks for duplicate tests in the set of 'Approved', 'Accepted', and checked-in tests; if found, reject the lesser-quality test as 'Duplicate', or suggest merging the two tests.

Reviewer checks that the test assertion (whether explicit or implied) is justified by the specification.

If the assertion regards functionality specified outside the css module indicated by the 'help' metadata links, reject as 'OutOfScope'.

If it addresses an ambiguity or open issue within the spec, set status to 'SpecIssue' and ensure that issue has been posted to www-style@w3.org or public-css-testsuite@w3.org and is being tracked appropriately.

Reviewer checks the test for correctness. (See the CSS Test Review Checklist for details.) If no problems are found, set the status to 'Accepted'.

Otherwise, Reviewer enters comments explaining what changes need to be made and sets the status to 'NeedsWork'. (If the Reviewer has the Author's permission to make changes directly, the Reviewer may also change the test as necessary. In this case, the status is set to 'Resubmitted' and someone else, possibly the Author, must review.)

Workflow for Approver

Approver searches the submission database for tests in the 'Accepted' state.

Workflow for CVS Monkey

If the test is now in the 'Approved' state, anyone with write access may now check it into the W3CCSS Test database, renaming the file as appropriate, and setting the status to 'CheckedIn'.

Feature Requirements

Check off X tests and give them all the same comment

Find test by ID, URL, title, etc.

Handle renaming of test

Track changes in same stream as comments (every subversion checkin adds a comment pointing to diff and new version)

Track status of test (needs review, needs work, etc)

Accept comments

Be test suite aware (which test belongs to which test suite); note that a test may belong to multiple test suites

Opt-in to hear comments on tests you reviewed (and accepted but second-level review found problems with?)

Easy to set up an account

Bulk submit preserving folder structure

Should work on all browsers

Unified login with wiki and Subversion

Backend Design Notes

A major design principle is to integrate with Subversion so that changes in SVN get automatically reported to the review system. Another is to take advantage of the tests' metadata so that as much information as possible will be automatically pulled from the test source.

Switching status to Approved automatically moves test to approved dir and marks as checked in

And so if you search Approved status, it searches approved directory

You can mark an Approved test as NeedsWork or Wrong: this svn copies it into the submitted directory and tracks it there. It keeps its checked-in status so we know this is high priority as it's live. Once re-approved, it gets svn copied back into the approved directory, overwriting the old.

Tests never lose their checked-in status unless they're deleted from the test suite