This document describes the formal schema of the Evaluation and Report Language (EARL) 1.0. The Evaluation and Report Language is a vocabulary to express test results. The primary motivation for developing this language is to facilitate the exchange of test results between Web accessibility evaluation tools in a vendor-neutral and platform-independent format. It also provides a reusable vocabulary for generic quality assurance and validation purposes. While this document focuses on the technical details of the specification, a companion document Evaluation and Report Language (EARL) 1.0 Guide [Guide], describes the motivations for EARL and provides an introduction to its use.

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This 3 March 2009 Editor's Draft of the Evaluation and Report Language (EARL) 1.0 Schema is an update of the previous EARL 1.0 Working Draft of 23 March 2007. It meets the requirements specified in the Requirements for the Evaluation and Report Language (EARL) 1.0, and incorporates change requests received since the March 2007 Working Draft. In particular, this draft implements the decisions of the Evaluation and Repair Tools Working Group (ERT WG) at its face to face meeting in November 2008 (see history of document changes). This document is intended to be published and maintained as a W3C Recommendation after review and refinement.

The Evaluation and Repair Tools Working Group (ERT WG) believes to have addressed all issues brought forth through previous Working Draft iterations. The Working Group encourages feedback about this document, Evaluation and Report Language (EARL) 1.0 Schema, by developers and researchers who have interest in software-supported evaluation and validation of Web sites. Feedback from developers and researchers who have interest in Semantic Web technologies for content description, annotation, and adaptation is also strongly encouraged.

Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

The Evaluation and Report Language (EARL) defines a vocabulary for expressing test results. It enables any person, software application, or organization to assert test results for any thing tested against any set of criteria. The test subject might be a Web site, an authoring tool, a user agent, or some other entity. The set of criteria may be accessibility guidelines, formal grammars, or other types of quality assurance requirements. Thus, EARL is flexible with regard to the contexts in which it can be applied.

EARL is not a comprehensive vocabulary for describing test procedures, test criteria, or test requirements but, rather, for describing the outcomes from such testing. EARL can be supplemented by test description vocabularies or other vocabularies for different aspects of the testing cycle.

A companion document Evaluation and Report Language (EARL) 1.0 Guide [Guide], to this specification provides more introductory material and explanation of the use cases for EARL. The companion document also highlights specific considerations, such as security and privacy.

The assumed audience of this specification is developers who are implementing EARL in software or processes, or those who are seeking to understand the ideas, models, or properties and classes used in the EARL vocabulary. Readers who would like more introductory material for the language with explanation of its foreseen use cases are referred to the Evaluation and Report Language (EARL) 1.0 Guide [Guide].

This document assumes that the reader is familiar with the ideas of Resource Description Framework (RDF) and can read its XML serialization. Readers who wish to understand more about RDF should read a general introduction or the RDF Primer [RDF-PRIMER].

Where an RDF term is used in its abbreviated form (e.g. Assertion or foaf:Person), that term is in the EARL namespace if no namespace is provided. The following prefixes are used in examples throughout this document:

[Editor's note: the following list will be revised.]

earl

The EARL namespace http://www.w3.org/ns/earl# which is described in this document

dc

The Dublin Core (DC) elements namespace http://purl.org/dc/elements/1.1/ whose terms are described in [DC]

dct

The Dublin Core terms namespace http://purl.org/dc/terms/ described at [DCT]

foaf

The Friend of a Friend (FOAF) namespace http://xmlns.com/foaf/0.1/ which is also where the terms are described [FOAF]

http

The HTTP Vocabulary in RDF namespace http://www.w3.org/2006/http# described in [HTTP]

This section describes the classes defined by this document. Every test result in EARL is expressed as an assertion. An EARL Assertion contains the following information:

Assertor

This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.

Test Subject

This can include Web content (such as Web pages, videos, applets, etc.), software (such as authoring tools, user agents, etc.), or other things being tested.

Test Criterion

What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.

Test Result

What was the outcome of the test? A test result could also include contextual information such as error messages or relevant locations within the test subject.

EARL provides flexibility to describes different types of assertions, such as those carried out by automated testing tools or by human evaluators, or those made about generic testing requirements or specific test cases.

Test Criterion - a testable statement, usually one that can be passed or failed. It is a super class for all types of tests including things such as validation requirements, code test cases, checkpoints from guidelines such as Web Content Accessibility Guidelines [WCAG], or others.

Manual - where the test was carried out by human evaluators. This includes the case where the evaluators are aided by instructions or guidance provided by software tools, but where the evaluators carried out the actual test procedure.

[@@@ tools can claim to "Support EARL 1.0" if they can partially produce or process EARL 1.0 reports. however, they can not claim to "Conform to EARL 1.0" until they can meet all the requirements of a producer or processor. we also need to do something about 'sensible' partial conformance. that is, not having tools that output nonsense or bogus reports like an assertion without an assertor.]

EARL is the result of the work of many people over the past. The editors would particularly like to thank Wendy Chisholm, Sean B Palmer, and Daniel Dardailler, whose contributions have included editing the first versions of the EARL specifications, and Leonard Kasday who set the work in motion to develop EARL. The editors apologise for any names left out of this list, and will endeavour to rectify any errors noted in comments.