Editorial changes to align with final REC SVG
test suite public release.

Preface

This document is intended as a permanent reference document and user manual
for designers and contributers to the SVG Conformance Test Suite. It
is the repository for currently agreed methods, templates, procedures, and
techniques. This is the third public release of the document, and correlates
with the public release of the completed BE test suite, subsequent to REC
SVG publication. The first release (mid-2000) corresponded to an earlier public
release of a subset (slightly over half) of the full BE suite. The second
public release (early 2001) was a full BE suite, and corresponded to public
CR draft of SVG.

Caveat

Some parts of this document remain incomplete. In particular, the synopses
of technical content of other conformance projects, in the chapter "Related
Conformance Work", have not yet been detailed. In addition, some procedures
and methods documented in the first public release have now been superseded
or deprecated.

1 Overview

1.1 Goals

The ultimate goal is a comprehensive and detailed conformance test suite
for SVG 1.0.

Amongst the multiple purposes which such a suite can serve, we identify the
most important as: a publicly available suite to help implementation
builders achieve interoperability.

1.2 Scope

There are at least three areas in which conformance testing is applicable:

conformance of SVG document instances;

conformance of SVG generators;

conformance of SVG interpreters and viewers;

This project's scope is limited to the third -- a conformance test suite
for interpreters and viewers.

1.3 What Kind of Suite?

At the Cupertino SVG-WG meeting (11/99), the question of what sort of suite
we are building was discussed and resolved. The options included:

An SVG demo suite;

A QA test suite for product developers;

A publicly available conformance suite and interoperability aid;

A certification test suite for a rigorous certification service.

The SVG WG decided at Cupertino: we are building #3, a publicly available
suite for such uses as informal conformance analysis and developer self-testing.

Presently, there are no plans for a SVG "certification" service,
therefore there is no need for #4 -- W3C doesn't currently carry out
certification testing, nor is any currently proposed by other entities.

1.4 Why Does It Matter?

What is the difference? In what ways would the suite differ depending
on its purpose?

Some potential differences include:

a demo suite doesn't really test anything precisely or in detail, it
is more of a "marketing" tool.

a QA suite would test goodness as well as correctness.

a certification suite often has aspects of formality, authority, and
"defensibility" which are not otherwise needed (i.e., outside of the context
of a certification service).

an interoperability aid might (should) test features in the standard
which are optional or recommended, whereas a strict certification suite
would not.

While the formality and rigor of a certification suite might not be needed,
the SVG conformance suite will (eventually) embody "traceability" (see below)
-- what specification in the standard justifies a given test?

1.5 Milestones & Schedules

The SVG WG agreed to two milestones:

delivery of a Basic Effectivity (BE) suite covering all of SVG 1.0, concurrent
with SVG's attainment of status of REC.

completion of a substantial body of detailed, drill-down (DT) tests.

At the time of release of this document, the BE suite is complete and the
SVG 1.0 REC has been published. Some initial work has been done on DT tests,
but it has not yet been integrated into the test suite structure, nor in this
current (REC) test suite public release.

1.6 Roadmap to this Document

For those interested in a quick user guide for test construction, you can
skip directly to "How to Write Tests".
The rest of this document provides background, detailed explanation, and motivation
for the methods used.

Section 3: "Graphics Testing Overview", contains a brief review
of principles and best practices learned, from the previous graphics suites
(CGM & VRML) as well as others (XML, DOM, CSS).

Section 4: "SVG Conformance Suite Development", describes all of
the design, approach, organization, and methodology which we've agreed so
far for the SVG suite.

Section 5: "How To Write Tests", a cookbook
type approach for builders of test materials. You can skip straight
to here, if you're building tests and you don't want the high level overview
of the design and methodology.

Section 6: Test Review Guidelines

Section 7: Glossary

Section 8: Bibliography

1.7 Document Status

The material in Section 2, especially the brief synopsis of the nature and
content of each existing suite, remains incomplete.

Section 3 is substantially complete.

The material in section 4 is complete for Static Rendering, but Dynamic has
not been addressed. A couple of topics like overall test-suite linking
structure are still incomplete (e.g., when substantial DT tests are generated
and added) .

Section 5 -- How to Write Tests -- is substantially complete for Static Rendering
"how-to", including incorporation of experience from a couple years' work
on the BE test suite.

Section 6, Test Review Guidelines, is complete.

Section 7, Glossary, contains a useful subset of key terms used in this document.

2 Related Conformance Work

2.1 Motivation

There is now a substantial body of test suite experience and material, for
several different standards:

the graphics standards CGM and VRML;

the W3C standards XML, DOM, and CSS.

These suites and the experiences of building them are useful to the SVG conformance
effort, and to contributors of test materials, in a number of ways:

guidance in overall architecture and techniques;

data for estimating the size of the SVG suite and likely level of effort;

some materials may be usable almost directly, or with minimal modification.

The "level of effort" data should be particularly interesting to the SVG
group.

Note. Since the initial release of this document (4/2000) to this release
(10/2001), there has also been substantial test suite work on XSLT, XSL-FO,
DOM Java binding, etc. These could be researched and include in a future version
of this document section.

2.2 Applicability of Previous Conformance Work

2.2.1 Previous CGM Work

See [5] and [3].

The applicability of CGM test suite experience to at least the static rendering
subset of SVG is obvious.

They are both 2D graphics languages,

and they have significant functional overlap in static graphical output
capabilities (CGM V3 is roughly equivalent to PostScript Level 2).

CGM and SVG differ in other ways:

CGM is static -- no associated DOM or animation concepts.

The industry profiles for which CGM test suites have been built are less
rich in "graphics arts" features -- color profiles, filter effects, compositing
and masking, etc.

CGM does not have def/use functionality like SVG.

Linking functionality is more limited in CGM.

CGM object hierarchies do not support composed transformations and interitance
of properties and attributes (or, more precisely, such CGM functionalities
are proscribed by the profiles).

Note. As of this date (10/2001), work is substantially complete and a public
release is pending for a test suite for REC WebCGM.

2.2.2 Previous VRML Work

See [9].

[Analysis of potential applicability VRML suite and methods will be written
for future document release.]

2.2.3 CSS

See [7].

For application of visual properties (graphical attributes) SVG borrows heavily
on CSS2. The syntax for all such properties (plus some others) is patterned
on CSS, and a number of CSS2 properties (esp. font properties) are adopted
directly by SVG. The full font selection and matching machinery of CSS2
is required in conforming SVG processors (interpreters and viewers).

W3C has made a test suite for CSS1. The methods of the CSS1 test suite should
clearly be applicable to some aspects of the SVG suite. Some actual
CSS test materials might be (almost) directly usable.

2.2.4 DOM

See [6].

We'll have to test the SVG DOM. NIST's XML DOM suite for Javascript
binding is released, for both XML and HTML. The Java binding is in progress.
Methods and techniques should be applicable, and maybe some materials can
be borrowed with minimal modification.

2.2.5 XML

See [8].

A conforming SVG interpreter (hence also a conforming SVG viewer) "must be
able to parse and process any XML constructs defined in [XML10] and [XML-NS]."
A conforming SVG viewer therefore incorporates XML-suite conformance, by reference.

2.3.4 XML

Level of effort: 1.5 FTE -- 2 people for approximately 9 months.
One person designed the test harness and some of the tests and the other
designed some tests and spent lots of time validating and filling in the holes
from what other people contributed.

Note. Current release (10/2001) is about 2,000 tests. Level of effort has
surely increased significantly, by an amount tbd.

2.3.5 DOM

Size of 1st release (Ecmascript with XML) -- 800 tests, 30,000 lines of code
(this is only the Fundamental and Extended tests).

Level of Effort: 1.5 FTE -- One person half time for 9 months, who
did the test harness (following much of what was done for VRML and XML); plus
another full time for about 9 months; plus a third person about 4 months full
time.

2.3.6 CSS

Unknown. To be researched for inclusion in future version of this document.

2.4 Synopsis of Design & Content of Other Suites

2.4.1 CGM

The CGM suite ([5]) consists of 269 test cases, each of which has three components:

There is no interactive harness or driver, and hence there are no navigation
buttons to assist navigation through the suite. The operator has to
invoke the viewer, access the Reference Picture, and access the Operator Script.

2.4.2 VRML

[Synopsis of content and structure of the VRML suite will be written for
future document release.]

2.4.3 XML

[Synopsis of content and structure of the XML suite will be written for future
document release.]

2.4.4 DOM

[Synopsis of content and structure of the DOM suite will be written for future
document release.]

[Javascript, XML and HTML finished. Java version being built now.
NIST did the Javascript XML, then Javascript HTML reusing some the data file
(which is a big HTML or XML document on which the DOM tests work). And the
re-used the Test Assertions and XML document for the Java DOM-XML tests.]

2.4.5 CSS

[Synopsis of content and structure of the CSS1 suite will be written for
future document release.]

3 Graphics Testing Overview

3.1 Process of Building Test Suite Content

The following basic process is applied for construction of most of the test
suites referenced above -- CGM, VRML, XML, DOM, at least. In overview:

Analyze the specification for testable assertions. These are called
"Test Requirements", TRs. (Note: other literature
refers to these variously as Semantic Requirements, SRs, or Test Assertions,
TAs).

Write a "Test Purpose" (TP) for each test. A Test Purpose
is simply: what we want to test -- the conditions, requirements,
or capabilities which are to be addressed by a particular test instance.

Write and document a Test Case (TC) which realizes each Test Purpose.

In practice, these steps need not be overly formal. In the case of
a certification suite, formality is important. For a conformance suite,
it is less so. In any case traceability (see below) is required.

Therefore, explicitly or implicitly, these steps are carried out --
the document is read exhaustively and decisions are made about what to test
about each functionality, and how to realize these decisions in a set of test
cases.

Section 4.2 of reference [9] contains an interesting discussion of TRs (which
it calls SRs) and TCs -- the step of generating TPs is implicit in this
reference, not explicitly treated as a formal step.

3.2 Principles Applicable to Test Suite Content

Some basic principles have been learned during the construction of previous
test suites, applicable to both graphics suites and others:

Simple or Atomic. Each test purpose should be as simple as possible
and narrowly focused on a atomic (simple) functionality. Example:
chose one attribute or property and exercise it through a range, while holding
other variables constant. The advantages to this approach are:

It is easy to see the cause, when an implementation fails (because
only one variable is being manipulated).

The suite can span all of the functionality of the specification.

The size of the suite will be manageable -- it will combinatorially
grow out of control otherwise.

Reducing number of tests. Without sacrificing the principle of
atomic testing, the number of test instances (files) can be reduced by having
a single instance combine multiple related test purposes. Example:
for an attribute or property with a half-dozen different enumerated values,
test each of the values in a "sub-test" of a single test case (test file
instance). Counter-example (poor practice): the first CGM test
suite sometimes had each instance test one value of an attribute, so that
almost 30 test file instances to test the horizontal and vertical text alignment
values.

Progressive. For any functionality, the tests should be organized
from easy and general to harder and more specific. This avoids wasting time
and resources if the implementation is completely incapable in a functional
area.

the simplest tests are called Basic Effectivity, BE;

the harder are called Detailed Tests, DT (also called drill-down).

Comprehensive. The detailed tests should try to methodically vary
and test all values, plus boundary conditions and extreme conditions, of
each parameter, attribute, or property.

Self-documenting. The tests should be self-documenting. For
example, a line-width test should have something like tick-marks drawn to
delimit the correct width. Graphical (displayed) text should explain
and/or label the pieces of the picture.

Key combinations. #1 notwithstanding, there should be some number
of tests which do vary more that one attribute or property at once.
Especially, thought should be given to how implementations might fail.
Example: CGM has separate but equivalent attributes for lines and
edges of filled primitives. Since it is common for implementations
to use a single stroke generator for both purposes, it is sensible to test
that the state of line/edge attributes are properly saved and restored after
drawing an edge/line.

Real examples. The CGM V3 suite included some real world
graphics arts and technical pictures.

these are called Demo tests, DM.

Traceability. A test must be traceable back to a statement or statements
in the standard's specification.

4 SVG Conformance Suite Development

4.1 Modularization & Prioritization

The SVG specification divides fairly cleanly into semi-independent functional
modules. Test materials will be developed and released progressively,
subject to the constraint that we have agreed to make an entire breadth-first,
basic effectivity (BE) test suite release first (now finished), and a drill-down
(DT) release subsequent to that.

Dynamic, including DOM, declarative animation, scripting, and all requirements
associated with "Conforming Dynamic SVG Viewer", including the High-quality
variant.

Static rendering got first priority for development and release of BE tests,
although work on dynamic proceeded in parallel, and followed closely on the
static tests. Prioritization (if any) for future DT work is tbd.

4.1.2 Progressive Ordering for Static Rendering

Functionality will be ordered in the suite, for purposes of execution and
navigation through the suite, from most basic to most complex -- implementations
should encounter the simplest and most basic tests first, before being subjected
to progressively more complex and advanced functionality. This mirrors the
organization of the REC SVG document.

Given our intent to make progressive releases of test suite modules, it made
sense to generally follow this ordering for the building of the materials,
at least for the completion of the DT and ER tests.

The SVG WG agreed at Cupertino (11/1999) to divide up the functionality by
chapter. For static rendering, the following chapters are addressed
in BE testing:

3--Rendering Model

4--(n/a)

5--Document Structure

6--Styling

7--Coordinate System, Transformations, and Units

8--Paths

9--Basic Shapes

10--Text

11--Painting: Filling, Stroking, and Marker Symbols

12--Color

13--Gradients and Patterns

14--Clipping, Masking, Compositing

15--Filter Effects

17--Linking (maybe not applicable to all static interpreters?)

20--Fonts

21--Metadata

23--Extensibility (?)

Building and executing tests in chapter order does not appear to always lead
to a basic-to-complex ordering.

Example: testing Rendering Model requires lots of other stuff -- in fact,
stuff from most of the other chapters.

Example: the geometry of paths, basic shapes, and text logically should
be tested before detailed testing of styling of these graphics elements.

From most basic (or fundamental -- basic does not necessarily mean simple),
to most advanced, a rough functional ordering might be:

The issue of test suite ordering and organization was resolved for the completed
BE suite -- chapter order. The issue is still open for future DT work.

4.1.3 Dynamic Module Prioritization

The decision was implicitly made for the dynamic parts of the BE suite --
chapter order, first priority assigned equally to the tests in a breadth first
set touching on each of the dynamic functionalities.

16--Interactivity

17--Linking

18--Scripting

20--Animation

Appendix B--DOM

4.2 Test Case Materials

4.2.1 Static Rendering Materials

Each Test Case (TC) in the static rendering module contains three principle
components:

.svg file -- the SVG instance, designed to test one or more Test Purposes;

The operator script comprises a few sentences describing what is being tested,
what the results should be, verdict criteria for pass/fail, allowable deviations
from the reference image, etc.

#1, #2, and #3 are file instances, one for each Test Case. #3 includes navigation
information, and is the source for generating the (HTML) navigation page.

Note. In the earliest test suites, for CGM, the Operator Script was
a rigid and rote checklist for use by (non-expert) certification testing technicians,
to score each test. It has evolved in more recent conformance work,
to being more informative about the test's purpose and what to look for. It
also can (and should) function to improve the accessibility of the test suite.

Details and examples of writing an Operator Script are given in the next
chapter.

Other supporting material are generated for each test case:

link(s) to the spec, providing traceback to the test requirements which
are being addressed;

test harnesses -- pages (HTML or SVG) which pull together the core test
components, and provide navigation through the suite.

Note. The traceability links were postponed until the SVG spec stabilized.
Now that REC has been published, this should be added to the test suite..

See below, section [4.4.2] and [4.4.4], for futher details about the test
harnesses.

4.2.2 Dynamic Module Materials

Most of the SR materials are applicable to most dynamic tests. However,
there may be cases (e.g., some DOM) which do not have graphical output, and
there will be some which could (but need not necessarily) have animated graphical
"reference images".

For the BE suite, the dynamic materials are the same as for static tests.
This material may be further refined as more of the dynamic functionalities'
tests are developed (looking forward to DT development).

4.3 Types of Tests

Four generic test categories have been decided. These are equally applicable
to static rendering and dynamic test modules:

Note. As of the date of this document (10/2001), only BE and DT tests have
gotten any implementation attention.

For BE tests, an attentive reading of the applicable spec sections is required,
but an exhaustiveTR enumeration is not. The generic BE test purpose
is: correct basic implementation, including major variations within
the functional area.

For DT and ER tests, an exhaustive TR extraction from the SVG spec is necessarily
a part of the process, and test purposes are be derived from the TRs (see
next chapter).

Following is a list of generic test purposes for DT tests (ER also?):

correct geometry,

accuracy,

full support valid value ranges,

all valid variants,

limit & boundary conditions,

big or otherwise stressful cases,

degeneracies (geometric, numerical, etc).

The Generic Test Purposes provide a high-level checklist for the sorts of
test cases which should result from the analysis and test design of a functional
area. If any major categories are not represented, it may indicate
that some implicit or explicit requirements have been missed.

4.4 Packaging, Organization, and Presentation

4.4.1 Standalone and Browser Requirement

These requirements are agreed, at least for the static rendering module:

the tests must be usable for a standalone viewer,

the suite should be conveniently navigable and executable from a browser.

4.4.2 Test Harnesses

The VRML suite, as well as the CSS, XML, and DOM suites, employ an interactive
test harness (HTML page), which:

draws together the materials for each test case -- test case file instance,
reference image if applicable, and operator script -- on a viewer screen
for presentation;

and, guides the navigation through the suite, from case to case.

Unlike the VRML suite, which presents the test rendering and the reference
image side-by-side in one window, the SVG WG initially decided on a two-window
approach, for standard release harnesses:

to satisfy #2 above, an HTML harness that presents the reference image
(.png) and operator script in one window, along with navigation buttons
(next, previous, parent, child, toc);

A single side-by-side PNG-SVG comparison harness, written entirely in SVG
(no OS display).

4.4.3 Test Naming Convention

A strong naming convention for the materials is useful, both with the management
of the test suite repository, as well as with requirement #2 above.

Test names aare brief but informative. The name design is: chapter-focus-type-num.
'Type-num' is a concatenation of the test type -- BE, DT, ER, or DM -- and
its ordinal in the sequence of such tests -- 01, 02, ...

4.4.4 Harness Details

The primary test harness (static rendering, at least), is an HTML page which
identifies the test, invokes the PNG reference image, and presents the operator
script. The plug-in variant also presents the SVG beside the PNG.

Navigation buttons are provided to go back to a table of contents (and maybe
an index), to navigate laterally through BE tests, and to drill down to "child"
tests (from the BE level to the DT, ER, and DM tests), to go back up to "parent"
tests from the lower levels.

Per an early WG decision (11/1999) and subsequent discussions, the principal
HTML harness presented the operator script and the PNG reference image, but
did not assume a browser-invokable SVG viewer -- the test administrator had
to get the SVG image into another window, or onto a printer, or whatever was
appropriate. There was a companion SVG-only harness to assist this (with exactly
parallel navigation capabilities to the PNG-plus-Operator Script harness.)

With the current method for producing harness(es) -- XSLT stylesheet applied
to instances of a simple XML grammar which describes each test case -- it
is not difficult to produce multiple harness versions, including variants
such as PNG plus rendered SVG plus operator script, for browser-plugin SVG
viewers. These have now been included in the public release.

The first generation design of simple XML grammar for describing tests, and
the XSLT stylesheet for producing the HTML page, have been released.
A "manual" SVG template has been released as well -- see next chapter for
details.

Work was completed on a "second generation" of harness and template
tools:

automate generation of per-test-case SVG templates and merging in of
the test body content.

This evolved into a test suite editor's tool, for performing major upgrades
that involve modifying the standard template in a uniform manner. Because so
many exceptions arose, which had to get manual or special processing, this line
of tools has been largely abandoned in favor of simpler generic methods -- 'sed'
scripts and the like.

Originally planned for the second generation tools:

include all test case history, maintenance, logistical, and bookkeeping
information in the XML description instances for each test case.

enable generation of (HTML) reports, inventories, and other information
about the test suite, were being done entirely manually during development.

After some working group experience with the BE suite and processes, it was
decided that this sort of "make" overhead was undesirable.

4.4.5 SVG Template Details

Each test , at least for the static rendering module, is put into a standard
template. As just discussed, this was originally a manual process --
the test writer puts the test body content into the template, and sets the
routine information items in the template. The second generation tools mentioned
in the previous section relieved this requirement somewhat, as long as the
content was written in the appropriate coordinate range, etc.

Features of the standard template include:

Canonical Size of 450 x 450. The template has width and height
of 450, without units, and does not contain a viewBox element. This
configuration to be used for all tests except those which probe size, viewBox,
and related attributes and functionality.

Valid XML, in particular has a DOCTYPE which references the operative
SVG DTD.

A comment block which records such information as author, revision history,
etc.

<title> element with the test case name, per convention described
above. Example. path-lines-DT-04.

<desc> element, with concise (one sentence is ideal) summary of
the test purpose, i.e., what this test is exercising. Example:
"Test that viewer correctly displays path components up to, but not including,
a command containing an error."

These two elements appear in the SVG file after the SVG element and before
the actual test content itself.

[Test body content is generated and inserted after <title> and
<desc>.]

Legend & frame. A block (<g>) at the end of the SVG file,
comprised of simple <text> elements so that they are part of the picture
-- with the following content:

identity of test suite and release version,

identity of the test case (same as <title>),

serial number -- a version control for the test case itself,

W3C copyright (hyperlinked to Document License).

#1, #3, #5, and #7 are the most critical. The template-generating tools do
carry these parts forward, and generate the other parts anew at each upgrade
run. An initial TC submission which has these parts can be input to the tools,
to generate a fully conformant SVG test case instance (the latter is a management
process for the "test suite editor", whoever that is).

The serial number is a method for ensuring that the PNG reference image,
the SVG instance, and the SVG rendering are all current, all agree.
To be useful, it must unfailingly increment whenever any change is
made to the SVG file, in fact whenever the SVG file is saved. Originally
this was a manual process. It has been replaced with the Revision keyword
of CVS (the suite is now maintained in a CVS repository).

4.4.6 Linking Order

The overall linking structure is: TOC, BE layer throughout the suite (next/previous),
DT drill down from BE (child), BE pop up from DT (parent), etc. Some
details are to be worked out. Does each BE point down to a different
DT "stack"? Or do all BEs in a chapter or chapter-focus point down to
the first DT in that area? The WG tentatively decided the latter (the
structure of the suite is not likely to be regular enough to make the former
widely practical).

An index of all files would be a useful addition, in some future release
of the test suite.

4.5 CVS Repository

Processes and procedures still need to be designed, and a test suite editor
assigned, for:

receiving contributions;

QA screening;

adaptation and integration;

incorporation into repository;

internal and external releases of the suite;

maintenance of tests in the suite.

Previously, the test suite editor was the repository, and all of these items
were ensured by the editor. The repository is now in CVS. An introduction to
it can be found in the document "CVS Repository for SVG Test Suite"
(presently only accessible to the WG). All WG members now interact directly
with the CVS repository. Experience has shown that there are varying degrees
of completeness, correctness, and integrity in the results with all-WG direct
access.

Once a test case was submitted, it is "owned" by the repository.
All WG and public releases are from the repository, and all maintenance changes
to test cases must be applied to the latest repository version.

If the second generation tools (for automatic SVG template generation) had
been made operational, adherence of test cases to the exact formatting (as opposed
to functional) details of the template conventions could have been somewhat
automated. However test contributors would have still had to adher to certain
minimum requirements in structuring the initial contribution (see previous section,
plus later sections about using the templates). And, as indicated earlier, there
were too many exception cases in which the automated procedures couldn't be
applied.

4.6 Sources for Tests

4.6.1 SVG Specification

The examples which already exist in the SVG Specification have proved to
be an excellent basis for material for some of the "Basic Effectivity" tests
-- simple tests which minimally illustrate an SVG functionality.

4.6.2 Vendor In-house work

Several of the SVG WG members have in-house development efforts. Materials
ranging from basic path and shape graphics, to filter effects tests, to DOM
and animation functionality are known to exist. Though some adaptation
and integration into the SVG Test Suite framework is required, these existing
QA materials have already proved to be a valuable resource for BE suite completion.

To be done: inventory what is available within the WG (for further
DT development -- BE is done).

Note. The OASIS XSLT/Xpath conformance committee has pioneered and is now
(10/2001) ready for public release of tools and procedures for accepting and
integrating collections of externally authored test cases. These should be
investigated for any future DT test suite construction.

4.6.5 Construct New SVG Tests

There is only one way to assure comprehensive coverage of Test Requirements:
build new tests, carefully designed and targeted at specific TR(s).

There are two ways to approach this:

Using a SVG output driver for a graphic arts or other program (useful
for those cases involving lots of graphical stuff and not requiring precise
control of the particular SVG elements and structure).

Manually (probably required for those tests which require very precise
configuration of structure and the order and selection of elements).

Experience with the first has been: the output drivers usually don't
have precise enough control of the individual pattern of elements, and manual
touchup is almost always required.

4.6.6 An Intersting Source of Test Purposes

From CGM experience, an interesting source of Test Purposes, and possibly
even test materials, are instances which have:

caused interoperability disputes,

and/or have led to non-trivial questions of interpretation of the standard.

These indicate trouble areas, where implementers are likely to misinterpret
or incorrectly implement the specification.

4.6.7 Other X*L Tests

Beyond static rendering module, it should be possible to leverage methodology,
or tests, or both, from such resources as the DOM test suite, [6]. CSS,
[7], should be applicable as well (for future DT test generation).

5 How to Write Tests

5.1 Overview

This is meant to be a cookbook for writing the test cases for a functional
module. Functional modules have generally corresponded to chapters in the
SVG spec. These techniques were prototyped with the Path chapter, and
have been applied in the generation of BE-level tests for the whole spec.

Note. When we did the Path chapter, an exhaustive TR extraction was
one of the first actions. This is not necessary prior to BE test case
specification, and the TR analysis is postponed until after the BE test case
generation in what follows.

5.2 Outline of BE Test Generation

Implicitly or explicitly, formally or informally, in this order or another
which you prefer, you will go through these steps for writing BE tests:

Read the document, all sections related to your functional topic.

Enumerate a list of the major pieces of functionality.

Decide about focus sub-sections, the focus sections for your chapter.

Figure out a set of test cases which covers the functionality list.

Derive a list of test purposes (at least implicitly).

Combine them into a convenient number of test cases.

Write the BE test case specification for each of the test cases.

For each BE test case instance, fill in the required information in the
SVG template.

Write the SVG code to implement the BE test case specifications, put
it in the template, refine till you're happy with it.

Write the XML description file, especially the Operator Script part.

For each BE test case instance, generate the HTML harness (for your own
use only).

Generate the PNG reference image.

Put it all together and tune it till you're happy with it.

5.3 Outline of DT Test Generation

DT test generation involves more rigor and more systematic methodology than
BE test generation. The basic steps for DT tests are similar,
with some differences at the beginning:

Read the document exhaustively, critically looking for any and all testable
assertions.

Enumerate a list of all Test Requirements (TRs).

Design a set of Test Cases which cover all of the TRs:

at least implicitly, derive a set of individual Test Purposes which
covers all TRs, with attention to the list of Generic Test Purposes;

combine these into a reasonable number of Test Cases of reasonable
complexity.

Derive a set of individual Test Purposes which covers all TRs, with attention
to the list of Generic Test Purposes.

Write the DT test case specification for each of the test cases;

The rest follows as for BE tests. The difference between the DT outline
and the BE outline is in the rigor and thoroughness of the first steps, which
are deciding "what to test".

Focus is simple to specify in some cases: shapes-rect-...; path-lines-...;
filters-feColorMatrix-...

In some cases, focus might not seem obvious. However, the "focus"
component of the name is always required.

"{BE|DT|ER|DM}" indicates that exactly one of the two-letter test
type designators is to be used, BE or DT or ER or DM. The numbering runs consecutively
throughout the chapter, it does not restart with each focus subsection.

Note. Use "camel case" for compound words for chapter and focus.
The first letter is lower case, and the first letter of subsequent words is
upper case. Examples: gradPatt, radialGradient, textAnchor.

5.5 Designing a BE Test

Remember, at the BE level, we are only trying to verify that the interpreter
or viewer has implemented the given functional area. Therefore, we focus
on the major functional pieces of the chapter.

There are no firm rules as to what comprises a BE test and what is DT --
sometimes it's a judgement call. However, here are some helpful guidelines:

the BE tests for a chapter should touch on all of the major functionalities
in the chapter,

taken as a whole, the BE tests for a chapter should give a good sense
that a viewer has implemented all of the major functional aspects of the
chapter,

but BE tests should not dwell in too much detail on any functionality,

BE test might reasonably exercise all of the values of an enumerated
or boolean attribute, if the number is small,

but otherwise, BE tests should not try to exhaustively probe a functionality.

Example. The 'path' element has a "d" attribute, which can contain
a number of commands: Mm, Ll, Zz, Hh, Cc, ... It also has the
"nominalLength" attribute, which is unique to Path. The BE tests for
path give a basic exercise of these attributes and commands, including verification
that the implementation understands the concept of subpath (holes and islands).

Note that there are also attributes like "style", "class", "transform", which
are functionalites widely applicable to other parts of SVG. It is a
judgement call, but in the path BE tests, we avoid these details -- they will
in fact be attacked in their own modules, such as Styling, Transform, etc.
(In other words, we don't deal with them extensively in the Path tests, not
even in the DT tests.)

We adopted a principle for the Path tests: just enough styling -- basic
colors, etc -- to make the tests visually less grim (than b/w, one-pixel wide
lines, no fill). This principle is applicable for all tests -- do not unnecessarily
clutter the test with functionalities which are unrelated to the functionality
being tested.

When starting to design the BE tests, look at the existing very simple examples
in the SVG spec for starters.

5.6 Writing BE Test Description -- Test Case Specification

The guidelines in this section apply to DT, ER, and DM tests, as well as
BE tests.

The test description is only for you, the test designer, unless you're dividing
the labor into specification versus production -- different people doing each
-- which we actually did on the Path prototype. Nevertheless, experience
shows that the challenge of writing the description force one to actually
design the test in sufficient detail that it is easy to then write SVG content
to implement it.

It is the Conceptual Description of the test (see below) which is useful
at this early stage (and it might be done after you have sketched/drawn the
Test Case).

In the Path prototype, the following format proved useful in describing Test
Cases:

Name (<title>): using our convention, e.g., "path-curves-BE-04".
This will be the root of associated filenames for the test case (TC).

Test Purpose (<desc>): a brief one-sentence summary of what
this TC addresses.

Operator Script: Summary of test purpose(s), how to execute test,
what to look for in result, some specifics about what constitutes pass/fail,
allowable variations from reference picture, etc.

Associated Test Requirements: Formally or informally, this information
should be available by the time the test case is designed (it is clearly
applicable to DT and ER tests, less so for BE and DM tests).

Document References: Links or pointers from the test to the
SVG spec.

See later section for writing the Operator Script (#4).

The Associated Test Requirements (#5) and Document References (#6) are the
crux of traceability. You will have generated this information (implicitly,
at least) by the time you have designed and written your test case. If you
are writing test cases, generate and preserve this information (see, for example,
next section and [11]).

See the next chapter, "Test Review Guidelines", for a concise summary
of other details which you should pay attention to, when you design and write
a test case.

Note.Traceability data currently are not integrated into the completed (REC
SVG) BE suite. This will have to be done (as links into the SVG spec) in a
future test suite release.

5.7 Extracting Test Requirements (TR) for DT
Tests

A comprehensive list of Test Requirements is the critical first step in writing
a comprehensive set of DT test cases for your functional area.

There is no magic rule. See [9], section 4.2, for an interesting discussion
of this. See [11] for a fairly thorough (as far as it went --
it's incomplete) example for the Path chapter.

It starts with an intensive reading of each sentence of your chapter.
Weigh the question, phrase by phrase: is there a testable assertion
here? If so, highlight it and add it to your list (however you want
to manage it. You also will need to read some other chapters, at least:

Error Processing;

Implementation Requirements;

Conformance Requirements;

and maybe others like Accessibility. Depending on your chapter, you
might be led off into other sections of the document for requirements, or
into other standards (e.g., CSS2).

Build up a list of TRs -- testable assertions -- which might
be applicable to a viewer or interpreter. In fact, don't descriminate
by "applicable to" at this stage. The SVG spec is written to describe
a file format, and the semantic requirements associated with data elements
are not always explicitly stated. For example from the Path chapter:

"The command letter can be eliminated on subsequent commands if the
same command is used multiple times in a row."

This is a statement about allowable data configurations within 'path' elements.
But combined with the statement form Appendix G, "All SVG static rendering
features ... must be supported and rendered ...", a testable assertion about
viewers results (and a test purpose can be derived).

You'll be lucky to find any (or many) statements which jump out with a "shall"
or "must" -- it doesn't occur once in my (incomplete) list of 59 TRs
for Path, in [11].

So for a first pass, pick up anything and everything which looks like it
might lead to a Test Requirement on an SVG viewer.

For the Path chapter, I assembled a list of Test Requirements after the first
intensive reading, during which I did markup on paper. You can follow
this sort of process, or do whatever is most agreeable for you.

Each entry in the list contained a document reference, and the text of the
TR -- I did cut-and-paste against the HTML version of the document for
the latter (note: there is some danger of volatility with this, at this
stage of the document).

Simple example:

Reference: 10.3.2.Mmtable

Statement: (x,y)+ -- Mm must be followed by one or more x,y pairs.

Lengthier example:

Reference: 10.4.p1.b2

Statement: nominalLength= The distance measurement (A) for the given
'path' element computed at authoring time. The SVG user agent should
compute its own distance measurement (B). The SVG user agent should then scale
all distance-along-a-curve computations by A divided by B.

This leads to another point. The general conformance requirements of
Appendix G imply a list of "Generic Test Purposes" (see [4.3]):

correct geometry,

accuracy (within 1 pixel),

full support valid value ranges,

all valid variants,

limit & boundary conditions,

big or otherwise stressful cases,

degeneracies (geometric, numerical, etc).

Keep this list at hand while you are looking at your TR list and deciding
what to test, i.e., deciding Test Purposes. (This list might be extended
-- suggestions welcome.)

I have been a bit informal with my "TP list", but I still keep track of what
I have covered on the TR list and the generic TP list, so that I know when
I'm done. See, for example, [11], the section, "Detailed
Drill-down Tests (DT) for Line Commands."

The final principle here, once you have an idea of what you're going to test
and how, is to put a reasonable number of "atomic" tests together to make
a Test Case. This will reduce the number of individual test cases and
increase their content density. The guiding principles should be:

maintain atomicity of the sub-pictures, i.e., their focus on concise
Test Purposes;

if a Test Case is getting too busy or big, then split it into two or
more Test Cases (pictures).

5.9 Generating the Template and Harness

The first release of a simple XML grammar for describing tests, and the XSLT
stylesheet for producing the HTML page, featured:

SVGTestCase.dtd -- the DTD for the simple XML grammar for describing
a test case.

These were the extent of the "first generation" production tools.
If you process the XML instances with CreateHTMLHarness.xslt (#2), you will
get HTML pages which pull together and presents the PNG reference images,
the operator scripts, and navigation buttons for the suite. Various batch
commands have been supplied as well, to assist in making one or more of the
harness types for one or a list of test cases.

Subsequent to the initial release of these tools, #2 - #4 have been rewritten,
factored to improve maintainability, and augmented with another harness --
all-SVG which presents SVG and PNG side-by-side (but with no OS).

These tools are all in the CVS repository, in the tools subdirectory.

If you process the XML instances with CreateSVGHarness.xslt (#3), you will
get a parallel set of SVG pages with SVG elements for navigation buttons,
and inclusion by reference of the test case SVG instances themselves.

If you process the XML instances three times respectively through the three
XSLT stylesheets of #4, you will get a set of three HTML files which generate
frame-based pages. PNG is displayed side-by-side with SVG (the latter via
a plugin), and the Operator Script is displayed below, in the bottom frame.

Initially I used only the XT tool of James Clark (get it from his Web site).
You can use whatever tool you prefer, but a caveat -- different XSLT processor
may give inconsistent results. Since the rewriting and revision of the
stylesheets, and addition of the 4th harness (all-SVG), I use Apache/xalan
for the two all-SVG harnesses (to avoid an anomoly involving appearance of
unneeded namespace declarations).

A set of DOS batch command files exists, to facilitate generation of the
harnesses. These are documented elsewhere (TBD).

The scheme has now been developed to the point that automatic generation
of the SVG skeleton file, with some of the details filled in (see next section),
is now possible. A set of advanced stylesheet tools was designed to do this
from a slightly expanded XML grammar (featuring things like 'desc' strings,
elements for author and creation date, etc). The final second-generation tools,
however, stayed with the simple initial grammar and don't automate all of
the required information. Therefore the static-output-template.svg is still
the place to start if you are writing a new test case.

5.10 Using the Template to Write Test Case

Starting with the static-output-template.svg and:

put the test name into the <title> element.

put a one brief sentence test purpose description into the <desc>
element

modify the preamble comment block

fill in the Legend block

write or otherwise generate the SVG test content and put it in the body,
where indicated (test-body-content).

Note about #5: this is critical. All critical content should be inside of
the "test-body-content" group. The second generation tools and maintenance
processes rely on it --most of the stuff outside of this group is automatically
generated, and it wouldd be lost if you put anything important there (the
'desc', and comment preamble will be preserved). Though these (2nd gen) tools
are not in production, it would be wise to keep the option open.

Note about #2. That the test should be self-documenting implies that the
picture should have a <text> element equivalent to the <desc>
(across the very top is a good location, in most cases).

Use good and thorough comments in the SVG content itself to describe what
everything is doing.

There may be some test purposes (e.g., the structure-emptySVG-BE-01.svg test)
which require no graphical content, in which (only) cases the Legend may be
omitted.

See below about the Serial Number (now superseded with CVS's Revision keyword).

See the next chapter, "Test Review Guidelines", for a concise summary
of other details which you should pay attention to, when you design and write
a test case.

5.11 Writing an Operator Script

The key feature of the XML instance is the Operator Script. Note when you
fill in your XML instance, you will also define the name (per convention documented
herein), and navigation links. The latter define the next and previous TCs
in the sequence for link navigation through the suite.

Be sure that your TC's XML instance links correctly to its TC neighbors,
and vice-versa! This is probably the most common mistake amongst
contributors -- failure to adjust the links of neighbors when a new test case
is added (or a test case is removed). (This is sufficiently annoying that
an automated verification or even correction tool is on "the queue",
for some future development.)

The Operator Script comprises a few sentences and is written as one or more
paragraphs of the XML instance (see earlier section) for the test case.

Once again, there are no firm rules. However, the Operator Script can
address any or all of:

describing what is being tested, i.e., a summary of the test purpose(s);

what the results should be;

verdict criteria for pass/fail;

allowable deviations from the reference image;

how to execute test, if there are any special instructions;

optionality of features;

prerequisites (other functionality used in the test);

accessibility aid.

#2 could conceivably be: "picture should look like the PNG".
However, some specifics could be pointed out, such as (for an accuracy test):
"All lines should pass through the cross-hairs", or "Vertexes should be at
the locations of the markers".

#3 and #4 go together. If there are allowable deviations of the
rendered SVG from the PNG, it should be stated (e.g., maybe a style falls
back to the default style sheet, which can vary).

About #6, optionality. If a test is exploring an optional or recommended
feature, that should be clearly indicated right at the beginning of the operator
script.

#7 refers to a brief description of SVG functionalities, other than the one
under test, which are used in the test file instance.

In addition to the other purposes of the Operator Script, a well-detailed
Operator Script can be useful as an aid to accessibility (#8).

5.12 Generating the PNG

Note. The following describes first-generation methods for getting the
PNG reference images. New tools are available from several sources, e.g.,
Adobe, CSIRO, and the Apache/batik project, which allowed direct generation
of correct PNG files for all test cases in the second public release, i.e.,
the complete BE suite.

When you develop a test case, you submit to the repository: the PNG reference
image; and, a description of how you generated the PNG.

5.12.1 Screen Capture SVG Rendering & Postprocess

By far, the most common method of generating the PNG reference image is:

Produce SVG.

Display the SVG with an SVG Viewer, such as the Adobe plug-in, the
batik viewer, etc.

Do a screen capture (e.g., Alt-PrtSc to clipboard in Windows);

Paste the screen capture into a tool such as Adobe ImageReady, Corel
Photopaint, Macromedia Fireworks(?).

Edit to trim away non-picture part, resulting in 450x450 pixel image.

Depending on the capabilities of such tool:

save as PNG (e.g., you can do this from ImageReady).

or, save in the native some portable format of the image editor
and load that into a PNG generation tool (e.g., Adobe ImageReady 2.0,
Corel Photopaint, Macromedia Fireworks), and save as PNG.

5.12.2 Screen Capture & Postprocess "Patch" File

The screen capture method relies on having an SVG viewer which can correctly
display the picture. Sometimes, especially in the early days of viewer implementation
development, this is not possible -- no SVG viewer handles the test instance
correctly.

However, it is often the case that there is another SVG file which is exactly
equivalent (pictorially). These are called "patch" files, and are
named, for example: structure-nestedSVG-BE-02-patch.svg (actual example).

Example: viewer doesn't correctly establish the origin of the user space for
a simple test of nested SVG elements. Then compensate for the viewer error by
changing the coordinates of the innermost graphical elements so that the viewer
positions them (graphically) correctly.

Example: viewer defaults something wrong. Then compensate by explicitly setting
that value (assuming that the viewer does this correctly).

Example: multi-stop gradients don't work, but two-stop are correct. Make a
"-patch" file where the correct picture of a multi-stop gradient is
built by stringing together multiple two-stop gradients.

Test contributors should submit any "-patch" files, along with the
PNG files and "how to" description.

5.12.3 Handcode the Reference Picture in a Graphics Program

This technique was used by some contributors, before the development of SVG
viewers was very advanced:

Produce SVG, e.g., by hand-coding.

Draw out the picture in a graphics program (e.g., Adobe Illustrator,
CorelDraw, or Macromedia Freehand) that shows how the SVG should look, save
as EPS (or another good exchange format for subsequent editing)

Convert to SVG with a SVG-to-CGM transcoder (e.g., one such is available
from IBM);

Take the body-content and put it into the SVG template, doing any hand
editing that might be required;

Render the CGM, screen capture and proceed as in previous methods.

Whether or not this will work for your test case depends on whether the result
of step #3 is close enough to the SVG configuration you need for the test
(and correct!), and more importantly, whether the hand-editing of #4 preserves
the graphical accuracy of the rendered picture.

A variant, for simple test cases, would be to hand-code the desired SVG,
hand-code (graphically) equivalent clear text CGM, and not use the transcoder
at all -- just use hand-coded CGM as a route to a correct picture.

This would only be useful, in place of the previous methods, if none of the
existing SVG viewers could get a correct rendering of the desired SVG test
case, and it was too difficult to reproduce the desired drawing in a graphics
program.

5.12.4 Notes about PNG Generation

Three aspects of the PNG file generation should have your attention:

file should be 450x450, matching the SVG test case.

file should use PNG-8 for compactness, unless PNG-24 is required.

avoid artifacts from color management systems.

The only exceptions for #1 are test cases which specifically deviate from the
canonical 450x450 coordinate space, in order to test viewer handling of different
SVG address spaces. In this case, match the SVG test instance.

8-bit PNG should suffice for most tests -- 256 colors are possible. 24-bit
PNG is likely to be required for tests such as:

gradients,

imported raster content, via 'image' element,

color range tests,

some animations.

While it might be possible to compute the number of colors required in some
of these cases, and optimize with PNG-8, nevertheless it is strongly recommended
to be conservative and use PNG-24, if there is any doubt.

For these same cases which require PNG-24, it has been discovered that attention
must be given to the color mode of the monitor, if screen capture is being used.
On PC Windows systems, for example, noticeable color banding has occurred on
some tests when using "High Color" (16-bit) mode, and it disappears
if "True Color" (24-bit) mode is used.

Avoidance of color artifacts is not yet completely understood. However, be
aware that color management systems on your computer may lead to incorrect colors
in the PNG reference image. One commonly seen example is a faint pink tinge
to areas that should be white. These color management artifacts can sometimes
be detected by using tools in raster editors (such as Adobe ImageReady 2.0,
Corel Photopaint, Macromedia Fireworks.)

5.13 About Serial Number

During CGM test suite development, a major annoyance and quality impact arose
from not being able to keep synchronization between the reference image (the
PNG files, for this SVG test suite), and the test case (SVG for us).
Changes made to the latter often weren't reflected by updating the former.
Worst of all, there was no way to detect the problem when it occurred.

A "serial number" in the SVG, which is encoded in graphical text, is the
solution for this -- it is quick and easy to determine if the PNG corresponds
to the SVG file and a given rendering of the SVG file (e.g., printout or screen
image) ... assuming that the PNG was generated from the SVG!

The serial number is part of the Legend of the SVG file. Previously,
the serial number was manually maintained and updated, which proved to be
something of an annoyance. Nevertheless, the version control benefits warranted
the inconvenience. Originally, the serial number was identical to a version
number -- 1, 2, 3, ... Its maintenance was solely the responsibility of the
test suite editor, which somewhat alleviated the error-prone manual aspects.

Now, the serial number text string (in the Legend of test cases and templates)
has been replaced with the Revision keyword of the CVS system. Every 'commit'
of a changed SVG file, no matter how trivial the change, automatically updates
the serial number. It is identical to the revision number in CVS.

Note: a very few .SVG files need to be treated as binary instead of text
(e.g., ones with wide range UTF-16 character codes). Presently (10/2001),
there are only two such test cases (still true?). Keyword substitution is
suppressed for these. Therefore the serial number must be manually typed into
the Legend after the rest of the test case production, and should match the
current CVS revision number.

6 Test Review Guidelines

6.1 Motivation for the Guidelines

There are two reasons that these guidelines are provided:

They comprise a brief synopsis of the most critical details, from the
previous chapter, for test developers;

And, it is an operational principle of this project that no tests will be
published until they have had review by someone other than the test author.

"Safeness" of test design -- this is vague, but ought to include such
attributes as being relatively immune to confusion if viewer fails, e.g.,
labels being obliterated by incorrectly rendered stuff.

Note. About "Self-documenting (i.e., in rendered content)," the
style of in-picture animation should be to describe what is being tested,
but should not describe visual effect (the latter may be done in Operator
Script). So for example,

correct: "linearGradient from the viewport left edge (red) to viewport
right edge (blue)"

incorrect: "You should see red on the left blending smoothly to blue
on the right."

6.4 XML Test Case Description

Specifically, this refers to the Operator Script, which is to be evaluated
for:

completeness & sufficiency;

usefulness;

correctness;

prerequisites;

allowable exceptions.

In addition, evaluate the correctness of the navigation links:

'previous' and 'next' point correctly to neighbors in the navigation
sequence;

vice-versa, the neighbors point correctly to this TC.

6.5 PNG Reference Image

Evaluate at least these criteria for the PNG reference image:

Does it accurately represent the correct picture for the SVG file?

It should be 450x450 pixels. Is it?

8 bit or 24 bit PNG:

8 bit suffices for most. Is it 8 bit?

If more than 256 colors are required for correct display, it should
be 24 bit. Are more than 256 colors colors required for correct display,
and is it 24 bit?

If generated from a "-patch" file, does the patch file define
a correct SVG equivalent for the correct rendering of the actual test case
SVG file?

It should be free of color management artifacts and effects.

7 Glossary

7.1 Basic Effectivity Test (BE)

A test which lightly exercises one of the basic functionalities of the SVG
specification. Collectively, the BE tests of an SVG functional area (chapter)
give a complete but lightweight examination of the major functional aspects
of the chapter, without dwelling on fine detail or probing exhaustively. BE
tests are intended to simply establish that a viewer has implemented the functional
capability.

7.2 Demo Test (DM)

A test which is intended to show the capabilities of the SVG specification,
or a functional area thereof, but is otherwise not necessarily tied to any
particular set of testable assertions from the SVG specification.

7.3 Detailed Test (DT)

Also called drill-down tests. DT tests probe for exact, complete, and correct
conformance to the most detailed specifications and requirements of SVG. Collectively,
the set of DT tests is equivalent to the set of testable assertions about
the SVG specification.

7.4 Drill-down Test

See Detailed Test.

7.5 Error Test (ER)

An Error Test probes the error response of viewers, especially for those
cases where the SVG specification describes particular error conditions and
prescribes viewer error behavior.

7.6 Semantic Requirement (SR)

See Test Requirement.

7.7 Test Assertion (TA)

See Test Requirement.

7.8 Test Requirement (TR)

A testable assertion which is extracted from a standard specification.
Also called Semantic Requirement (SR) or Test Assertion (TA) in some literature.
Example. "Non-positive radius is an error condition."

7.9 Test Purpose (TP)

A reformulation of a Test Requirement (or, one or more TRs) as a testing
directive. Example. "Verify that radius is positive" would be
a Test Purpose for validating SVG file instances, and "Verify that interpreter
treats non-positive radius as an error condition" would be a TP for interpreter
or viewer testing.

7.10 Test Case (TC)

As used in this project, an executable unit of the material in the test suite
which implements one or more Test Purposes (hence verifies one or more Test
Requirements). Example. An SVG test file which contains an elliptical
arc element with a negative radius. In practice (and abstractly), the
relationship of TRs to TCs is many-to-many.

7.11 Traceability

The ability, in a test suite, to trace a Test Case back to the applicable
Test Requirement(s) in the standard specification.