Test suite planning

Norm looks for help and advice in constructing
an adequate test suite on the assumption we'll make it through
Last Call and get to CR easily.

Norm: There are already about 80
or so at tests.xproc.org

Henry: What's the metadata you
need?

Norm: There's documentation for
the test format at tests.xproc.org, but I'm happy to adjust it
as necessary.
... We will stall if we don't get a test suite...

Henry: My metadata question was
about coverage. We need to give some thought to the metadata to
describe *what features* are tested.
... We need to be able to characterize the tests to make the
interoperability claim have some teeth.

Norm: I was going to analyze the
test suite for missing steps and missing options on steps.

Henry: That's a good start, but
there are also questions about the topology of the pipeline and
the inheritence of variables and defaulting of pipes,
etc.
... Those aren't as easy to enumerate.

Norm: I was also hoping to have
tests for every static and dynamic error as well, that covers
many of those things.

Richard: The XML test suite
doesn't attempt to identify the "correct" error and this has
sometimes caused problems.

Henry: Having explicit error
codes helps a little bit.

Norm: Metadata is a good idea,
maybe we should have some email discussiona bout how to devise
metadata that covers the spec.

Any other business?

Vojtech: Regarding error codes,
right now we have gaps in the numbering. Is that ok?

Norm: I think it is. We talkeda
bout it once before and decided that the cost was larger than
the benefit.