Current Situation

The SVG 1.1 test suite must be manually run, visually looking at each test SVG.
This includes tests of the DOM that merely set the fill of a rectangle to green or red to indicate that a test passes or fails.

By convention, a test that shows red indicates a failed test.

Tests include data in the private namespace 'd'. The data includes a test description, instructions for running (if interaction is required), and the pass criteria.

There is only limited testing of internal CSS style sheets.

Types of Tests

Reference tests

SVG vs SVG Ref

Test SVG can be compared pixel by pixel with a reference SVG.
These types of tests could be automated.

Tests of CSS styling vs. presentation attributes.

Tests of illegal syntax.

SVG vs PNG Ref

Test SVG cannot be compared exactly with a reference SVG.

This includes tests where renderers can produce different results due to (allowed) rounding errors, etc. (recall the SVG specification allows a one pixel error).

ED: Note that another complication might be that errors can accumulate so that the one pixel error margin may not enough by itself.

Tests with filters.

Tests with gradients.

Techniques

Inkscape at one point had a regular testing program.[[3]] The program relied on comparing a newly generated image against known good and bad images. By default only a binary diff was done. There was an option to use an external program to do the comparison. perceptialdiff was the suggested program.[[4]]

Non-static Tests

Tests of Animation

Tests of SMIL

Tests of CSS Animations and Transitions.

Question: Can the results of animation via SMIL be compared to CSS Animation via a ref test or via JavaScript (querying the final positions)?

The suggested format has been designed to minimize the differences between SVG and CSS testing to facilitate adapting the CSS test framework to our needs.

Multiple links with the same "rel" are allowed (e.g. one can have multiple authors).

The CSS Test Format page gives clear instructions on file names. Maybe we should adopt the same naming conventions.

The CSS framework provides a set of shared resources in the support directory.

Issues:

We will need a new NS URL for SVGTestCase.

Not needed with a template adopted below.

Do we need to keep the copyright section? The CSS tests do not include a copyright.

Include via link.

The CSS that the CSS test is testing is included in the <head> section where it can easily be found. This idea would be nice for SVG tests that test CSS so the CSS can be displayed along with the test (example). However we cannot put our CSS in the same place. We need a style section outside the SVGTestCase block with an appropriate 'id'.

Do we need to keep the revision displayed in the SVG or can we rely on Mecurial?

CSS has a meta section for "flags" to indicate what features are needed by the renderer for the test. Two flags relevant to SVG are: "svg" and "namespaces". Since SVG is SVG, do we need to include these flags or can we just assume that any SVG renderer will support SVG and namespaces?

The flags allow SVG tests to be used by test suites for other specs. Also not all SVG test, for example tests of SVG in HTML, can be run by all SVG renderers. We should include them.

Do we need a frame?

No, one can use a border to offset the SVG when including the SVG in an HTML test harness.

Structure after 8 March 2012 telcon

Most significant change is to remove <html> wrapper as this will cause problems for SVG inside HTML. Note <meta> causes HTML5 parser to auotclose the SVG, reverting to HTML, while <link> does not.