olivier Thereaux wrote:
>
> Dear all,
>
> some of you may remember an earlier mail thread [1] on this list, where
> we discussed a list of test cases validome is using to compare their
> tool to other validators. I thought at the time (and still do) that this
> was an excellent list, but that I would like to be able to see
> authoritative sources for what the expected result would be, making the
> list more of a test suite, and less of a marketing tool.
>
> [1]
> http://lists.w3.org/Archives/Public/www-validator/2006Apr/thread.html#msg67
>
> Having a test suite for conformance testing tools is extremely
> important, for a whole slew of reasons:
> - Specification are not always perfect and it is sometimes difficult
> for conformance testing tools to interpret them.
> - Often, conformance to a specification goes well beyond validity
> against a grammar. Sometimes, there isn't even a grammar
> - Some specifications don't have test suites. When they do, they often
> are useful to test what is supposed to work. Conformance testers are in
> the realm of dealing with what doesn't conform, what doesn't work. In
> other words, most test suites are focusing on other classes of products
> than testing tools, and testing tools need their own test suites.
>
> In discussions with Alex of Validome, we talked about this topic, and
> through these discussions, the idea to make a framework that would
> manage test suites for markup language validators and conformance
> checkers was born.
>
> The basic idea is to create a repository of test cases that would make
> it possible to create simple test cases: a document, an expected
> conformance checking result, and references/argumentation for the
> expected result. If input to the framework can be collectively
> contributed to, even better. Of course, a repository of test isn't very
> useful if it can't be fed to a system that can run the tests (e.g [2])
> and compare the output of one (or more) testing tool(s) to what is
> expected.
>
> [2] http://dev.w3.org/cvsweb/2002/css-validator/autotest/
>
>
> The Validome team, I think, is already starting work on this, and the
> result will be open and open source. This would be a great opportunity
> for cooperation, and I am sure that the participants in this list who
> are also developing and maintaining such tools (Hello Christoph, Jirka,
> Nick, Sam, Henri etc.) would be primary customers for it, and would be
> interested in participating - either in the development of the test
> framework, or in the building of test suites for testing language X or Y.
>
> I have already invited the validome team to get cvs accounts here at
> W3C, which I think would be a good place for such development, as a good
> meeting ground, and for the insurance that resources hosted here will be
> persistent and public. I would like to extend that invitation to others
> interested in participating in this project. Please contact me if you are.
I'm interested.
- Sam Ruby