This document describes a methodology for evaluating the conformance of websites to the Web Content Accessibility Guidelines (WCAG) 2.0. It provides guidance on defining the evaluation scope and parameters, exploring the website features and functionality, sampling representative web pages where it is not feasible to evaluate all web pages of a website, applying the WCAG 2.0 success criteria and conformance requirements in this setting, and documenting and reporting the evaluation findings. It extends the existing guidance for WCAG 2.0 but it does not replace or supersede it in any way.

This methodology only addresses evaluation of a website after its development, for example to validate its conformance with reasonable confidence or to understand its accessibility performance to improve it. It is applicable to any website, including web applications and websites for mobile devices. It is independent of any particular evaluation tool, web browser, and assistive technology, and it addresses different contexts, including self-assessment and third-party evaluation. However, ensuring and maintaing accessibility requires that accessibility is considered throughout the development process. Complementary resources from W3C/WAI on evaluation provide guidance for other situations.

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This 3 September 2012 Editors Draft of Website Accessibility Conformance Evaluation Methodology 1.0 is a complete draft based on the same approach and expanding the previously published W3C First Public Working Draft of 27 March 2012. This Editors Draft addresses the following comments received, to seek approval for publication as an updated Working Draft:

This document is intended to be published and maintained as an informative W3C Working Group Note after review and refinement. The WCAG 2.0 Evaluation Methodology Task Force (Eval TF) invites discussion and feedback on this document by developers, evaluators, researchers, and others with interest in web accessibility evaluation. Eval TF is particularly looking for feedback on:

In general we are also looking for feedback on the terminology used in this version. The term "Methodology Requirement" used in this document is temporary.

Please send comments on this Editors Draft of Website Accessibility Evaluation Methodology for WCAG 2.0 to public-wai-evaltf@w3.org (publicly visible mailing list archive). These comments will be considered in the internal discussions of Eval TF.

Publication as Editors Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

Website owners, procurers, suppliers, developers, and others often need to evaluate the conformance of existing websites to the Web Content Accessibility Guidelines (WCAG) 2.0. For example, such evaluation may be carried out as part of releasing or purchasing a website to validate its conformance with reasonable confidence, or it may be to better understand and monitor the level of accessibility provided on a website for improvement. The methodology described in this document provides one possible approach to address this particular aspect of web accessibility evaluation.

The methodology described in this document supports evaluation in different contexts including self-assessment and third-party assessment of websites. It is applicable to all websites, including web applications, websites intended to be used with mobile devices, and other types of websites. It is independent of the size of the website, the web technologies used to create the website, and of any particular web accessibility evaluation tools, web browsers, assistive technologies, and other software tools.

This document provides guidance on defining the evaluation scope and parameters, exploring the website features and functionality, sampling representative web pages where it is not feasible to evaluate all web pages of a website, applying the WCAG 2.0 success criteria and conformance requirements in this setting, and documenting and reporting the evaluation findings. It extends the existing guidance for WCAG 2.0 but it does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way.

[Editor Note: This section has been added in this Editor Draft and needs review.]

WCAG 2.0 conformance requirements apply to individual web pages. However, on most websites it is not feasible to evaluate every web page with reasonable effort. To ensure conformance of a website to WCAG 2.0 it is necessary to ensure that this is considered throughout the development process. Following this methodology helps assess the level of conformance with reasonable confidence where it is not feasible to evaluate every web page.

This methodology only addresses evaluation of a website after its development. It does not specifically explain evaluation in other situations nor does it provide general guidance on evaluation with WCAG 2.0. Complementary resources from W3C/WAI on evaluation provide guidance for other situations.

Website developers who want an evaluation during the development process.

People who want to use a methodology for education and outreach activities.

Policy makers, project managers, and others who need a standardized approach for evaluation.

Users of the methodology are assumed to have expertise in WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web; read more in section 2.2 Required Expertise.

The following documents introduce the essential components of web accessibility, inclusive design for people with disabilities and people with aging-related impairments, and overlap with design for mobile devices. The documents help managers, designers, developers, policy makers, researchers, and others optimize their efforts in these overlapping areas:

From WCAG 2.0 Conformance Requirement for Complete Processes: When a web page is one of a series of web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.)

Web pages that are relevant to the entire website. This includes the homepage, login page, and other entry pages, and, where applicable, the sitemap, contacts, site help, legal information, and similar web pages that are typically linked from all web pages (usually from the header, footer, or navigation menu of a web page).

Primary functionality of a website including tasks that users of a website carry out to perform this functionality.

Note: Examples of functionality include "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".

From ATAG 2.0 definition for "templates": Content patterns that are filled in by authors or the authoring tool to produce content for end users (e.g., document templates, content management templates, presentation themes). Often templates will pre-specify at least some authoring decisions.

[Editor Note: Given that ATAG 2.0 is still a Working Draft, this definition might change in future drafts of this document.]

A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and web applications.

Note:Websites may be composed of smaller sub-sites, each of which can be considered to be an individual website. For example, a website may include an online shop, an area for each department within the organization, a blog area, and other areas that may each be considered to be a website.

From WCAG 2.0 definition for "Web Page": A non-embedded resource obtained from a single URI using HTTP plus any other resources that are used in the rendering or intended to be rendered together with it by a user agent

The methodology defined by this document is used for comprehensively assessing the conformance of websites to WCAG 2.0. A more cursory approach for exploring the accessibility of a website is described in Preliminary Review of Websites for Accessibility. In many cases it is advisable to carry out such preliminary reviews before applying this methodology, for example to identify obvious errors and to develop a rough understanding of the overall performance of the website.

[Review Note: Eval TF is particularly looking for feedback on this section.]

The methodology defined by this document applies to full, self-enclosed websites. This includes websites of organizations and entities, persons, events, products, and services. It also includes web applications and mobile websites. Examples include:

"The public website of Example Org. located at http://www.example.org"

"The intranet website of Example Org. located at http://intranet.example.org"

"The online shop of Example Org. located at http://www.example.org/shop/"

"Release version 1.5.3 of Online Reservations System (ORS) application"

"Dutch version of the Example Org. website located at http://nl.example.org"

A website may include several smaller sub-sites, such as an online shop, an area for each department within the organization, a blog area, and other areas that may each be considered to be a website. This methodology can be applied to make conformance claims for such individual sub-sites, in which case these sub-sites would be considered to be a website each, and to the main website in its entirety. However, this methodology may not be applied to a websiteexcluding any of its parts.

Example of a Website:

In the diagram above, the university website is made of several sub-sites: "Information for Students", "Information for Lecturers", a "Courseware Application", and a "Library Application". The "Courseware Application" includes "Physics Courses", "Medicine Courses", and "History Courses" that are aggregated into the application. Also, the university website has individual web pages such as legal notices, sitemap, and others that are not part of any particular sub-site.

In the example above, none of the depicted parts may be excluded from the scope of evaluation if a conformance claim is to be made for the university website according this methodology. This includes aggregated and embedded content from third-party sources such as online maps for the university campus and forms for credit card transactions. Excluding parts of the website from the scope of evaluation would likely conflict with the WCAG 2.0 conformance requirements full pages and complete processes.

In some cases only one or few web pages may dynamically generate all the content with the functionality that normally an entire set of web pages would provide. Such web applications are usually not separable into sub-sites and are considered as a single website or part of a website. Each state (individual representation of content and functionality) in which such a web application can be considered to be a web page for the purpose of this document.

In some cases websites may have clearly separable areas, such as a password-restricted area of a website (extranet) that is not part of using the public area (log-in is not required to complete a function or process). Such areas can be considered as individual websites rather than sub-sites for the purpose of this document.

Some websites are available in versions that are independent of each other. For example, mobile versions of a website and versions of a website in different languages, where using one versions does not require or depend on using another versions of the website (usually such website versions have a different set of URIs each). Such website versions can be considered as individual websites rather than sub-sites for the purpose of this document.

Users of the methodology defined by this document are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web. This includes understanding of the relevant web technologies, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and evaluation techniques and tools to identify potential barriers for people with disabilities. In particular, it is assumed that users of this methodology are deeply familiar with the resources listed in section 1.3 Background Reading.

The methodology defined by this document is independent of any particular web accessibility evaluation tool, web browsers, and other software tools. However, web accessibility evaluation tools can significantly assist an evaluator during the evaluation process and contribute to more effective evaluations. For example, some web accessibility evaluation tools can scan entire websites to help identify relevant pages for manual evaluation, and other tools can assist manual evaluation in a variety of ways. Specific guidance is provided in Selecting Web Accessibility Evaluation Tools.

Note: Web accessibility evaluation tools can only automatically check a limited set of accessibility requirements that are automatable. Conformance evaluation requires manual testing and judgment by experienced evaluators.

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise. However, using the combined expertise of review teams provides better coverage for the required skills and helps identify accessibility barriers more effectively. While not required, it is strongly recommended to employ review teams for conformance evaluation of websites. Specific guidance is provided in Using Combined Expertise to Evaluate Web Accessibility.

The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise. However, involving people with disabilities and people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by the evaluator alone. While not required, it is strongly recommended to involve real people covering a wide range of abilities during the evaluation process. Specific guidance is provided in Involving Users in Web Accessibility Evaluation.

This section defines the stages and activities of the evaluation procedure. The stages are not necessarily sequential but rather iterative. Also the exact sequence of the activities carried out during the evaluation process depends on the type of website, the purpose of the evaluation, and the process used by the evaluator. Some of the activities overlap or be carried out in parallel. The following diagram illustrates the iterations between the stages defined in this section:

During this step the overall scope of the evaluation is defined. It is a fundamental step that affects the subsequent steps in the evaluation procedure and is ideally carried out together with the evaluation commissioner (who may or may not be the website owner) to ensure common expectations about the scope of the evaluation. Some exploration throughout this stage may be necessary to understand the full scope of the website and the required evaluation (see 3.2: Explore the Target Website).

Note: Involvement of the website owner and/or website developer (in addition to the evaluation commissioner) is not required but often helps identify use cases, functionality, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.

During this step the exact scope of the website (the web pages for which the evaluation will apply) is defined and documented. This scope definition may not contradict the terms established in section 2.1 Scope of Applicability. It is essential to carefully document particular aspects such as any use of third-party content and services, different languages, and parts of the website that may not be easily identifiable as such (for example an online shop that is not integrated into the website but considered to be part of it).

The outcome of this step is an unambiguous definition for the target website, so that for each web page it can be determined whether it is within the scope of evaluation or not. Where possible, it is recommended to use formalizations such as regular expressions or listings of URIs that define the web pages that are within the scope of evaluation (part of the target website).

Only identifies whether a website conforms or not without providing additional information. This type of evaluation is typically carried out when the website is assumed to conform, for example to verify an existing conformance claim, and for large-scale evaluations with less resources to explore the details of individual websites.

Identifies whether a website conforms or not, and provides further information about the conformance of each evaluated web page. This type of evaluation is particularly useful for instructing web developers and for acquiring statistics for monitoring progress over time.

Identifies whether a website conforms or not, and analyzes any identified issues in detail. This includes descriptions of the errors and suggestions for possible repair options. This type of evaluation is particularly useful for organizations that want to improve the accessibility of their website and need a detailed analysis

Part of initiating the evaluation process is to define the target WCAG 2.0 conformance level ("A", "AA", or "AAA"). In many situations WCAG 2.0 Level AA is the generally accepted level of accessibility.

It is often not feasible for websites to support accessibility on every combination of web browser, assistive technology, and operating system that they run on, nor is it possible to test with every such combination of tools. It is necessary to determine the primary context in which a website is used and the minimum set of web browsers and assistive technology that must be supported by the website.

For example, a website may be public, only available within a closed network (intranet), or limited to authenticated users (extranet). Users of intranet and extranet websites may be known to have a certain level of training on using the website and may be using pre-determined web browsers and assistive technology. This is usually not the case for public websites.

Note: It is also not possible to single out or exclude individual groups of users, such as "people with visual impairments", in the target audience. All WCAG 2.0 Success Criteria must be considered throughout the evaluation.

W3C/WAI provides a set of publicly documented Techniques for WCAG 2.0. However, it is not necessary to use these particular techniques. In fact, in some situations, such as in a closed network, it may be necessary to use techniques that are specifically developed for such situations. Individuals and organizations developing techniques must employ methods for establishing the technique's ability to meet the WCAG 2.0 Success Criteria.

During this step the evaluator explores the target website to be evaluated, to develop a better understanding of the website use, purpose, and functionality.

Carrying out initial cursory checks during this stage helps identify web pages that are relevant for more detailed evaluation later on. For example, an evaluator may identify web pages that seem to be lacking color contrast, consistent navigation, or document structures (headings, links, etc.) with simple checks while studying the website, and note them down for more detailed evaluation later on.

To carry out this step it is critical that the evaluator has access to all the relevant parts of the website. For example, it may be necessary to create accounts or otherwise provide access to restricted areas of a website that are part of the evaluation.

Note: Involvement of the website owner and/or website developer can be helpful to help identify common web pages, functionality, technologies, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.

During this step the common functionality of the website are identified and documented, to provide a comprehensive basis for the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample. The outcome of this step is a list of user tasks such as "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".

Web pages with varying styles, layouts, structures, and functionality often have different implementations of accessibility features. They are also often generated by different templates and authored by different people. Web pages may also appear differently, behave differently, and contain different content depending on the user. The outcome of this step is a list of the different types of web pages on the website, to provide a comprehensive basis for the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample. Different web page types include:

During this step the web technologies relied upon to provide the website are identified and documented, to provide a basis for the sample selection through the steps defined in 3.3 Step 3: Select a Representative Sample. This includes base web technologies such as HTML and CSS, auxiliary web technologies such as Flash, Java, JavaScript, PDF, Silverlight and WAI-ARIA, as well as advanced web technologies such as SMIL and SVG. The outcome of this step is a list of web technologies that are relied upon.

Note: Where possible, it is often also useful to identify the libraries and components used to create the website, such as Dojo, JQuery, and others. Particularly for web applications, much of the accessibility support is built into these libraries and components, and evaluation can become more effective and efficient when these are identified.

While ideally every web page of a website is evaluated, usually this is not possible on most websites. In cases where all web pages can be evaluated, this sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.

Exploration of the target website in 3.2 Step 2: Explore the Target Website (within the scope set in 3.1 Step 1: Define the Evaluation Scope) provides sufficient understanding of the website to facilitate selection of a sample of web pages that is representative of the entire target website. Depending on the size and complexity of the website, the size of this sample will vary. For example, a website with few types of web pages that are all generated from a confined set of templates, such as a data repository, will likely require a smaller sample than a website with many types of web pages that are authored using different mechanisms. Also a web application could have a limited number of web pages that dynamically generate content with varying types of appearance, behavior, and information that each need to be sampled.

The web pages identified through the exploration will typically relate to more than one design aspect. For example, web pages with particular functionality such as scripting, multimedia, and forms will typically also use related technologies such as Flash, JavaScript, PDF, Silverlight, and WAI-ARIA, and in many cases these web pages may have different design to others. Careful selection of web pages can significantly reduce the required sample size while maintaining appropriate representation of the entire website.

Note: In this step the evaluator identifies the comprehensive sample of web pages for evaluation. However, many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Guidance on evaluating the sample identified in this step is provided in 3.4 Step 4: Audit the Selected Sample.

Note: A selected web page could have any number of these features. For example, a selected web page could be used to represent the use of forms and scripting at the same time. The important aspect is to select at least two distinct web pages for each relevant feature identified on the website, though more web pages may be necessary depending on the complexity of the website.

The selected sample must include all web pages that belong to a series of web pages presenting a complete process. Also, no web page in the selected sample may be part of a process without all other web pages that are part of that process to be also included into the selected sample.

Note: According to WCAG 2.0, Success Criteria to which there is no matching content are deemed to have been satisfied. An outcome such as 'Not Applicable' may be used to denote the particular situation where Success Criteria were satisfied because no relevant content was applicable.

To carry out an evaluation effectively, it is often useful to construct and apply personas, use cases, and scenarios of users with a variety of abilities and using different web browsing techniques, including assistive technology and adaptive strategies. It is critical to consider the broadest possible spectrum of use cases to help identify issues that may occur to different audiences. It is strongly recommended to also involve real users during this process, to help identify issues that may not be easily identified through expert testing alone. See Involving Users in Evaluating Web Accessibility for more guidance.

This assessment also needs to be complemented with focused testing of particular situations including:

Confirmations for input, error messages, and other feedback from user interaction;

Behavior of web pages using different settings, preferences, and interaction parameters;

Note:Templates are often used to create many web pages, sometimes entire parts of a website. While evaluating templates is optional in this methodology, in some contexts it can be helpful to check templates on their own. Evaluating templates may identify potential issues that may not be easily identified through evaluating individual instances of web pages. However, issues identified in templates alone do not necessarily imply that these issues occur on the website and need to be validated on individual instances of web pages. Also, identifying no issues in templates does not necessarily imply that no issues occur on instances of web pages.

Met when for each applicable instance of the WCAG 2.0 Success Criterion on the web page at least one Sufficient Technique is identified to be applicable, and no Common Failure is identified to be applicable;

Not met when for any applicable instance of the WCAG 2.0 Success Criterion on the web page at least one Common Failure is identified to be applicable;

WCAG 2.0 techniques are not exhaustive as they cannot cover every possible situation. Also, the techniques used to meet WCAG 2.0 Success Criteria during the development may not be known to the evaluator. Particularly for newly released web technologies or when these web technologies are used in particular contexts there may be no publicly or proprietary documented techniques available to the evaluator. The evaluator must be considerate of these limitations when using WCAG 2.0 techniques

In some situations WCAG 2.0 techniques and repositories on accessibility support provide insights on the level of accessibility support for accessibility features in particular combinations of web technologies and tools. However, the evaluator is responsible for the accuracy of the assessment of accessibility support and the resulting evaluation.

Archive web pages for future reference. At the very least, include the title and URI of the web page and the date of the evaluation. To avoid ambiguity, complement this with any of the following:

Copies of the web page files and resources; Note: Some tools can save the dynamically generated or modified content (DOM) available during the evaluation rather than the initial content of the web page that may be different.

For future reference, record Evaluation tools and methods used to evaluate the web pages. This includes versions of web accessibility evaluation tools, web browsers and add-ons, and other tools used during the evaluation. Depending on the level of detail for reporting defined by 3.1.2 Step 1.b: Define the Goal of the Evaluation, this recording may apply globally for the entire evaluation, to individual web pages, or to individual checks carried out within the audited web pages.

This section describes how to report the evaluation findings that have been gathered during the previous steps. Reporting your findings is a key element of every evaluation and helps facilitate replicability of the results.

Note: Individual pieces of the reporting are gathered throughout the evaluation process, not necessarily at the end of it.

Provides the same information as a detailed report with additional analysis of the successes and failures identified for the website. This includes a summary of the positive and negative aspects identified on the website, examples of frequently occurring issues and an assessment of their impact on the users of the website, and suggestions for improving the overall accessibility of the website.

Accessibility statements according to this methodology can also be made when only partial conformance has been achieved according to the requirements defined in third-party content and languages lacking accessibility support. In such cases the accessibility statements must also include information to identify the following:

Performance scores can provide more granular measures on the level of conformance of a website than the conformance levels defined by WCAG 2.0 provide. However, scores alone do not provide sufficient context and information to understand the actual level of accessibility of a website. Performance scores according to this methodology may only be provided when the evaluation carried out conforms with this methodology as per 5. Conformance with this Methodology.

Depending on the requested level of detail, the performance score is calculated through one of the following approaches:

Note: According to WCAG 2.0, Success Criteria that do not apply to the content are deemed to have been met. Web accessibility evaluation tools may be needed to help count and calculate the scores.

The approach used to calculate the score must be indicated together with the numeric ratio whenever the score is provided. For example, "X/Y Per Website", "X/Y Per Web Page", and "X/Y Per Instance" are valid statements to express the performance score.

Machine-readable reports facilitate processing the results by tools, for example to help monitor accessibility of a website over time. The Evaluation and Report Language (EARL) is a machine-readable format that was specifically designed for this purpose. It is recommended to use EARL for providing machine-readable reports. See also Understanding Metadata to learn more about uses of metadata, including machine-readable reports, such as EARL.

The methodology defined by this document is flexible to allow its implementation in different situations and contexts. This section provides guidance for applying this methodology in some of these situations.

Note: It is recommended to carry out preliminary reviews before carrying out a conformance evaluation to identify obvious errors and to develop a rough understanding of the overall performance of the website.

Evaluation of large websites according to this methodology may be broken down into the evaluation of individual website areas, such as the online shop, departments within an organization, blogging area, and other sub-sites. In this case, each sub-site must meet the terms established in section 2.1 Scope of Applicability and must be evaluated using this methodology. Evaluation of each sub-site must be carried out to at least the same conformance target (as per 3.1.3 Step 1.c. Define the Conformance Target) as for the main website.

The methodology defined by this document is flexible to allow its implementation in different situations and contexts. It is not required to carry out any of the steps and activities defined by this methodology in any particular sequence. However, it is required that the following Methodology Requirements defined by this methodology are met:

Editor note: A complete example template for evaluation reporting in human understandable language. Discussion proposal below is built upon the idea to have three different levels of reporting. The template will have to be ok for official evaluations so some change is still necessary. The difference between option 1 and 2 is currently very small (open for discussion).This section needs appending to the Methodology Requirements in section 4. this section may change in later drafts depending on how the other sections evolve.

This Appendix proposes three different levels of reporting following the discussion in clause 5 on levels of evaluation. The three examples are:

Example 1: Report to determine pass/fail

Example 2: Report including examples of resources where the fails can be found

Example 3: Report including examples of resources where the fails can be found and suggestions for repair

A concise description of the Web pages, such as a list of URIs (for which the claim is made), including whether subdomains are included (in the claim).

Provide a "list of success criteria beyond the level of conformance claimed that have been met."

Description of the website that has been evaluated including a general description of what was included and what was not included

A list of web content technologies relied upon and a list of those that are excluded from the evaluation (but available on the evaluated website).

Partial conformance claims (if this is an evaluation for conformance)

Results: per guidelines, checkpoint indicate pass/fail. Also give an indication of the frequency of the problem (global, regional)

Disclaimer(s)

Results

Guideline X (heading)

Checkpoint: SC XX (subheading)

Result: pass/fail

Character: global/regional (or another wording) if regional: a list of pages where the problem exists

Description of existing problems and barriers for users or link to descriptions on W3C/WAI website..

General information about this checkpoint: link to how to meet and to understanding this success criteria.

Action: Description of techniques for meeting the SC (could be techniques which are already in the techniques document or new techniques which are not in the document, but with which the SC can be met).

List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.