Table of contents

There is an agreed upon need to produce a highly comprehensive
conformance test suite (DOM TS) for each of the DOM Level specifications so far
specified and also for DOM Level 3. In order to accomplish this, it
has been agreed between W3C and NIST that these test suites be
jointly produced. W3C will be responsible for coordinating these
issues. NIST has agreed to allocate resources to update their current test suite
to full Level 1 specification support, taking into account the
alterations needed as indicated below. The test suites will be
jointly developed by these two parties and will take the form of a
public framework as explained below.

A test suite for DOM Level 2 will at a later stage be produced,
building on the test suite for Level 1 and drawing on the
experiences made while producing it, as well as experience drawn
from related work in this area. It is understood that the Level 2
specification is quite large and covers too big an area for one
single person to be able to write the entire test suite. For this
reason, among others, but also aiming at wide-spread acceptance as
well as contributions both from individuals and companies, it has
been decided that the test suite be publicly developed, and the DOM
WG suggests the design of both this test suite and the one for
Level 1 be as indicated below.

As far as the test suite for Level 3 is concerned, it has been
decided that it initially takes on the form of a small feature test
suite, not complete, and advances to more comprehensive and public
status similar to the test suites on level 1 and 2 once the DOM
Level 3 WD reaches CR status.

This document does not deal with technical issues, as these have
not yet been finally decided upon. There will be a document that in
detail explains the technical design of the test suite framework;
however it is appreciated that these questions will be of
interested to the community and in order to allow for input,
it has been decided to let this issue take top priority once
the interested parties have gathered to start working with the test
suite.

The DOM test suites should be developed in a public framework.
The development of this framework itself is a central area of
interest, in addition to the DOM TS in general. The DOM WG welcomes
the participation of interested parties in developing the test
suites. However, there are various issues, mainly procedural in
nature, that need to be resolved.

The DOM test suites are intended to be used as a tool to aid
implementors in developing DOM implementations. Validation and
certification of these implementations are outside the scope of
this work. The tests and test suites will be provided for
information and help only. However, we intend to produce as
comprehensive, functional and general tests as possible, and this
should be the overall goal in designing and implementing the DOM
TS.

The test suites could be hosted on the W3C site once developed.
Where the test suite should reside while being developed is still
being discussed. They could also after development be mirrored at
various sites in order to simplify access and enhance availability
to the community.

The DOM TS for all levels of the DOM will be produced in a
public framework with contributions from developers and companies
in the community. The DOM TS will be coordinated and supervised by
the DOM WG and NIST. The representatives will use resources from
their organisations as well as invite individuals and companies
(through representatives) to allow for a larger number of people to get
involved.

Since this will be a public framework, there are a series of
procedural issues that need to be resolved. There is a need to
have a stable mechanism for submitting tests to the test suite.
It's been proposed that we look into a multi-level process (with
the possibility of tests passing between the different levels for
reasons explained below). The points below assume that there is a
technically stable mechanism for submitting tests, saving
information about those tests, version handling and so forth.

The DOM TS editor will maintain a list of tests needed that will be available at the DOM TS development web site. It is presupposed that developers or organizations submitting tests have done a sanity check with regard to the functionality of the tests by referring to the relevant specification.

In order to simplify for individuals and companies
the production of the test suite and, if they so wish, contribute
with tests of their own, we propose the following process for
submitting tests:

The test, or series of tests, get submitted to the framework.
Submitter should also send
a notification to the mailing list that will be set up for the DOM
TS to indicate that he/she has submitted a test to the DOM TS
framework. Submitting parties are also encouraged to check the tests they are submitting against the list of tests needed.

It is acknowledged (presumably by the moderators/reviewers of
the test suite) that the test has been received. This should be
done via email directly to the submitter, and a copy should be sent
to the mailing list. The test receives the "received" status.

The test together with the necessary documentation is saved for
future reference.

The test suite's moderators investigate the test given the
following criteria. At each point is also indicated possible
reasons for not accepting to publish the particular test.

Relevance for the particular implementation of DOM (the
level should be indicated in the documentation). It can here be
decided that the particular test is irrelevant to the TS since it
tests a feature that is not in the specification.

Part of the specification being tested. If there are
other, previously submitted tests that are functionally equivalent
and test the same part of the specification, this test should either be
kept for archive purposes but not published if it is functionally equivalent to a previously submitted test, or take the previously submitted test's place in the test suite.

Quality of code and documentation (the code should be
stable and not contain any errors, the documentation should be
written in accordance with the XML file in which specification,
functionality and possible scenarios are documented).

Scenarios that underly the particular test layout and
its intended scope.

Development guide for the DOM TS, once developed. The existence of these guidelines will be communicated through the DOM TS mailing list.

If the test is decided to be stable, which means that it is
relevant, that it indeed tests a particular identifiable part of
the specification, and that it is not wrongly written, it becomes
"Reviewed and Stable" and receives this status.

Tests that at this or some other stage are judged not to be
appropriate for publishing receive the "Inappropriate" status.
These tests should be kept for archive purposes.

In cases where tests have received the "Reviewed and Stable"
status but are found to be erroneous, the following procedure is
proposed:

If it is found that a test that has been called "Reviewed and
Stable" actually is not stable (if the tests are not correctly
written or for any other reason), this particular test gets
submitted to the DOM WG for further consideration. Possible
outcomes of this stage is that the test, after consideration, is
judged to be

Correct, in which case it goes back to "Reviewed and Stable"
status

In need of further investigation and possibly rewriting (after
which it receives the "Reviewed and Stable" status)

Inappropriate for publication according to the previous
point.

If a test is judged to be correct after consideration, it keeps
the "Reviewed and Stable" status.

If a test as a result of this re-evaluation is judged to be
inappropriate, it receives the "Inappropriate" status.

We propose that each test or whole suite comes fully documented
with regard to the following (there will be an XML wrapper for this
kind of information available from the TS website):

Name of submitter, date of creation, and, if appropriate, company.

Suggested ID for the test (this may be changed by the moderator
to comply with the identification mechanism of the technical
platform).

Part of specification being tested. If the test is written to test ie. The
Node
interface in DOM Level 1, this should be suitably indicated, either by giving a
link to the relevant part of the specification (in this case
http://www.w3.org/TR/1998/REC-DOM-Level-1-19981001/level-one-core.html#ID-1950641247), or by
indicating, in text, that it is this part of the specification that is being
tested.

If appropriate, the scenario which has preceded the writing of
the test. This could take the form of a textual description, eg.
"This test investigates whether a user of XX can use a feature such
that so-and-so happens on mouse over".

The DOM Test Suites should consist only of submitted, reviewed
and stable tests.

The DOM TS for all levels will also consist of documentation on
the tests it contains, scenarios that have served as a base for
them, commented code and possibly comments from the editors.

It is proposed that the W3C DOM WG representative act as
moderator and controller for incoming tests. The DOM TS moderator is choeon by the DOM WG. All tests should
be kept for archive purposes, whether they get published or
not.

The DOM TS will be a publically developed suite of tests. As such, it seems plausible to assume that there will be some changes and revisions. In order to avoid confusion as to what test are being run, the DOM TS framework will consist of different versions of the TS. The DOM TS moderator will be responsible for versioning and proper inclusion of tests in the various versions of the DOM TS. DOM TS 1.1, for example, is different from DOM TS 1.0, and will be properly documented with regard to what has been changed (added, deleted, or otherwise altered).

The DOM TS aims, as explained above, at helping implementors to write applications that support the DOM specifications. In no way are these conformance tests in the sense of providing companies or institutions with certification of DOM support. The only claim that could be made is that a particular implementation is conformant to a particualr version of the DOM TS. There are two cases of results of running the test suite:

The implementation fails to pass the test suite. In this case it can be asserted that the implementation fails to meet the relevant DOM specification.

The implementation passes the test suite. In this case all that can be asserted is that the implementation is conformant to that particular version of the DOM TS.

It is understood that no submitting party, individual or
company, can claim ownership of submitted tests. All copyright
notes and similar text must be removed from the code prior to
submitting the test (or tests) to the DOM TS. It is also understood
that no submitter can try to withdraw submitted tests, but must ask
the TS moderator that the test be withdrawn. Reasons for this
should be given, for reasons of future reference. In this case, the
moderator should send a notification to the DOM TS mailing list in
which is stated that the contributor for a particular test wishes
to withdraw the submitted test. Only after that and after approval from the DOM WG can the test be
withdrawn from the DOM TS.