Quality Assurance (QA) Activity Proposal

1. Summary

W3C creates the technical specifications regarded by the Web community at
large as Web standards. In order for these standards to
permit full interoperability and access to
all, it is very important that the quality of implementation be given as
much attention as their development.

As of the beginning of 2001, W3C has published more than 20 Recommendations. As our family of
specifications gets more complex, their acceptance and deployment on the
market becomes an ongoing challenge. The past experiences of HTML, CSS or more
recently SMIL (all implemented with various degrees of conformance by vendors)
are strong incentives to start this Activity with due diligence.

A message
(W3C Member-only page) related to formalizing our work on QA at W3C was sent
to w3c-ac-forum@w3.org in 1999 by Daniel Dardailler. The initial feedback from
AC members on this list was very positive and led us to hire a Conformance
Manager (Karl Dubost) whose role would be to
organize and lead this Activity.

In April 2001, a W3C QA workshop was held at
NIST which again showed the interest of the W3C community in this area.

This briefing package proposes such an Activity, with a focus on
solidifying and extending current practices (existing
validation tools, test suites, common test framework, etc.) and
thinking ahead in terms of certification, education, funding,
and tracking the quality of products and services related to W3C
technologies.

We therefore propose the creation of a Working Group
focused on building better specifications and test tools and of an
Interest Group to share ideas on using these tools, building
metrics, and communication.

2. Context

There has always been and still is a strong demand from the Web community
(end users, Web agencies, organizations, software developers, etc.) for better
support and better implementation of W3C specifications in products
(commercial or not).

All Web users have at one point or another experienced the problem of
invalid pages rendered incorrectly on their platform or valid pages
rendered incorrectly by their platform.

Most experienced Web authors have at one point or another been faced with
the choice of either developing several versions of a page or picking up one
particular solution detrimental to a portion of their audience.

The overall mission of a QA team at W3C would be to improve the
quality of W3C specification implementation in the field. In order to
achieve that, we will need to

work on the quality of the specs themselves (e.g. make sure they have a
conformance section, a primer, clear text that is unambiguous for
developers, good layout, consistency between specifications - e.g. TAG relationship, etc.);

promote the development of good validators, test tools and harnesses for
implementors and end users to run;

think ahead in terms of what additional steps could be taken to achieve
this goal more efficiently (certification, education, communication,
etc.).

Fortunately enough, we're not starting from scratch in this area.

Several W3C Working Groups, as well as individuals from the W3C Team or
the Web community have started to assemble test suites, produce validation
tools and follow good QA practices. As a pre-Activity work item, we've been
gathering these existing W3C related QA efforts in a QA Matrix.
In addition, external organizations or individual persons, such as NIST or OASIS, have been active in the field of
conformance and testing of W3C technologies, with varying degrees of W3C
Working Group coordination.

These existing efforts are very important and will serve as a basis of
future work as we move forward in the formalization of our new QA Activity
at W3C. However we believe that in order to be really effective, QA work for
W3C technologies must be driven from inside W3C, with very
close coordination with W3C Activities, in order to give a unique and
approved reference for external developers.

QA is absolutely necessary in order to ensure interoperability and
usability and also to have consistency between the specifications we produce.
The QA Activity will benefit the Web community, industry, and Members, as a
guarantee of interoperable products, which is the core mission of W3C.

3. Proposal: Quality Assurance Activity

QA applies to all W3C Activities. The QA Activity itself will therefore not
be attached to a particular W3C domain but will exist independently as a cross
domain Activity. Daniel Dardailler will act as Domain Leader.

The Activity will initially be chartered for two
years.

Scope of the Activity

The Quality Assurance Activity is dedicated to all aspects concerning the
quality of the implementations of W3C specifications in
products (commercial or not) as well as the quality of the
documents we produce. W3C wants to improve the level of
interoperability of software used to access (read/write) and serve the Web.
The Activity should help the implementations of the W3C Recommendations with
tools such as validators and test suites. The Activity will strongly encourage
Working Groups and/or third parties to produce test suites. It should also
focus on the quality of documents produced at W3C, the consistency of these
documents, and to ensure levels of conformance are well defined in our
Recommendations.

QA Working Group

The mission of the QA Working Group is to organize and
unify the work done in W3C groups in building and designing test suites and
validation tools. It also defines the terms of QA, using a glossary, a
taxonomy file, and also creates "how-to" guidelines for building test
suites.

The main objective of the Working Group is to foster the development of
usable and useful test suites endorsed by the W3C, which share a common look
and feel, and ensure that the validating tools of the W3C are fully
operational, useful and educational.

As a step in performing this task, the QA Working Group will need to work
on conformance testing methodology, on the quality of the W3C specifications
with respect to conformance and clarity, and the tracking of issues related to
specification ambiguity and evolution.

QA Interest Group

The QA Interest Group's main objective is to have W3C, its
Membership and the Web community involved in QA at large share their
understanding of the state of affairs related to Web certification, branding,
educational effort, funding model, etc.

The list of activities foreseen in this Interest Group are:

share experience in validation of Web content, documents and
protocols;

discussion about QA position in W3C standardization cycle (from early
draft to Candidate Rec);

issue with external funding, partnering with commercial sector,
IPR;

how to address certification: branding, service, logo, metrics;

education: tutorials, documentation, etc.;

evolution of the Activity into a more comprehensive domain to guarantee
full compatibility between all W3C Recommendations and the unification of
the various groups doing specification reviews (like WAI PF, I18N,
TAG).

Certification and branding appears to be a key point in many organizations
and governmental agencies. It helps the customers to identify the tools that
produce documents of quality and it applies to browsers, as well as editors,
software libraries, Web servers, etc. As of the writing of this proposal,
certification is considered outside the scope of the Activity
but we'll need to clarify what a certification system should do, who should
run it, and how to interface with this market.

Although not a core Activity of W3C in general (except in the WAI domain), we think education and documentation are a
necessity to reach the level of quality we want to see in the implementation
of our standards. To start with, many Web developers, technical writers and/or
software developers have no idea what W3C is, what a W3C standard is, and why
it is important for them. The QA Activity will need to work in cooperation
with communication team to promote the awareness of standards, tools to check
them and the rationales for using them.

The following picture made at
the QA workshop uses concentric circles to describe the kind of work foreseen
in the Activity. The innermost circle (SPECS DEV: ASSERTIONS, REVIEWS, etc.)
describes the work on improving the specifications, which serves as a basis
for further test development. The next circle is about test development (TESTS
DEV: GUIDE, CHANGE CONTROL, EXT COORD, COMMON HARNESS). Together they form the
basis for the Working Group. The last circle gives ideas about the things we
should be thinking in the context of the interest group (TESTS USE: METRICS,
COMM, CERTIF).

4. Confidentiality and Intellectual Property Rights (IPR)

We expect the two groups and the Activity resources in general to be
publicly accessible.

All documentations, test suites, validating tools produced inside this
Activity will have to be defined under a license. There are questions about
the kind of license to use (for instance we have two licenses at W3C: one for
document and one for software, the document license being more restrictive for
the change control, which may be of interest to ensure the integrity of QA
tools).

We expect the tools to be freely usable, runnable, and downloadable in any
case.

Prior disclosure of intellectual property rights pertaining to QA applied
to W3C technology will be required, following the IPR requirements
defined in the W3C Process.

5. Participation and Resource Statement

W3C Staff Resource Commitment

In addition to this core team, which we expect to grow over the life of
this Activity, it's important to point that most of the W3C technical staff
acting as staff contact for a technology will end up working "part time" for
QA, when their technology is up on the QA agenda. E.g. X is not in the QA
team, but when the QA team starts looking closely at a test suite for the
specification he or she is dealing with, we expect X to free up some
time/resource to coordinate with us, interface with people in the group, help
us promote whatever W3C test guidelines we have come up with during WG
meetings, if needed, etc.

We also expect staff resource allocation to the Technical Architecture
Group (TAG) to be accounted as part of the QA
activity.

All W3C group participants should realize that QA is to be considered a
"natural overhead" of any WG, even if it's not written down in our process
yet. QA will succeed only if every person inside W3C participates in it.

External Participant Commitment (W3C Members or invited experts)

For participation in the QA Working Group, the requirements
for meeting attendance and timely response are described in the W3C
Process document. This participation (attending meetings, reviewing documents
and preparing drafts) is expected to consume between one half to one
day per week.

We expect and will be welcoming different communities to contribute to the
Activity:

W3C Working Group participants (staff or not) developing
specifications;

W3C Working Group participants developing tests for specifications;

W3C Team and other individuals working on validators/test suites;

Organizations that specialize in conformance testing for Web
technologies (W3C or not);