Important note: This Wiki page is edited by participants of the EOWG. It does not necessarily represent consensus and it may have incorrect information or information that is not supported by other Working Group participants, WAI, or W3C. It may also have some very useful information.

This file is protected (locked for editing) to provide a stable reference.(EOWG: This is an archive of previous comments. Put new comments in the WCAG-EM review page.)

March 2013 Comments on 26 February 2013 Working Draft

General

Table of Contents: Change headings to something like: "Contents Summary" and "Detailed Contents". Include the Appendices in the Detailed Contents list, rather than under a separate heading.

initial comment: I wonder if it is necessary to list the main section in an "overview" heading, then to list all sections in a "sections heading" and have a list of appendices at last. Why not displaying a classical "table of contents" as in WCAG or other W3C documents? May be give the ability to expand/collapse the different sections (in the table of contents)? {Sylvie 22/March}

additional exchange: I think an expand/collapse menu could work {Vicki - March 31}
This is to follow the W3C TR format, and I don't think it's worth pushing for adding the expand collapse here. OK? {Shawn}
Ok - cool with that {Vicki - 3 April}

Step headings and "Methodology Requirement":

Differentiate the Step headings from the statement that follows. Maybe make the headings even more terse, e.g.,
"Define the Scope of the Website"->"Define the Scope",
"Define the Goal of the Evaluation"->"Define the Goal",
"Define the Conformance Target"->"Define the Conformance",
"Define the Techniques and Failures to be Used (Optional)"->"Define the Techniques and Failures (optional)"

Delete "Methodology Requirement" and the colon so it's like:
<h4>Step 1.a. Define the Scope
<p class="req">1.a. Define the scope of the website.

initial comment: I don't understand the difference between h4: "Step 1.a: Define the Scope of the Website" and the following sentence: "Methodology Requirement 1.a: Define the scope of the website." When you listen to it, it sounds redundant but there may be a reason for this redundancy. I think this question is repeated over all the section. {Sylvie 22/March}

updated

Optional sections: Explain up front why some sections are optional, and that in most cases the optional sections are strongly recommended. If warranted, reiterate in specific sections that the step is strongly suggested and note why it's best practice.

initial comments:

Step 4c: Use Techniques and Failures should not be Optional. The section title already says "where possible." Backing away from established techniques and failures reduces confidence in the usefulness of the Techniques. Rewrite to emphasize that the Techniques allow for new ways to meet the SCs but should be followed in general.Rationale: For consistency, predictability of results, etc. Could also remind evaluators to add to the Techniques if they find previously undocumented ways to meet the SC. {Sharron 3 April}
There are reasons behind this. It's outside EOWG to debate the reasons; however, I think it is within EOWG scope to discuss how to effectively communicate the issue. {Shawn}

Step 4e Again I do not understand the reason for this being optional. Surely a conformance review should be replicable. Recent experience shows significantly different outcomes based on tools and testing environments. Tools documentation should be part of the report - period.Rationale: I cannot think of an example where the tools and platform are irrelevant to the testing results. If someone can think of such an exception, perhaps provide that as a Note. But in general, the tools etc are an important part of the result. {Sharron 3 April}

Basic Report, Detailed Report, In-Depth Analysis Report - Step 1b: Defining the Goal & Step 5.a.
Please come up with better terms for each and better explanations. Also, the main explanation should be in one place; it's confusing to have it in two places, as it currently is. EOWG started talking about these levels in in 5 April telecon and comparing the two explanations, but didn't finish. Perhaps we might have a joint discussion?

initial comments:

I am not sure I agree that there is sufficient difference in the two in-depth reports to require three categories. Seems clearer to stick to 2 - basic and detailed and maybe say a bit about the fact that the amount of detail is good to be part of the definition as well.Rationale: It may be splitting hairs but it seems that once you have made the decision to go beyond a statement of conformance, there is an entire spectrum of detail and nuance. Why not four categories then? or more?{Sharron 28 March}I thought the problem lay in the label; the 3 definitions seemed distinct but "detailed report" did not seem to accurately reflect the 2nd report according to its description since there's no detail. Perhaps "Statistical Report" or "Itemized Page Report" would be a better fit. "Basic Report" - not sure this fits the description either. Perhaps "Pass / Fail Rating" would be more appropriate.{Howard 4 April}

Eval TF note: Comments on the abstract also apply to the similar sentence in the main document.

Replace "one possible approach" with something that gives it more credibility and more strongly recommends it, e.g., "one established approach". see brainstorms in [approach - EOWG minutes 5 April]

initial comments:

Rationale: It sounds too tenuous and it seems we could be more encouraging. Something like “one proven method” or something. {Sharron 28 March}

Replace "However, this methodology does not replace the need for quality assurance measures that are implemented throughout the design, development, and maintenance of websites to ensure their accessibility conformance."
with: "This methodology does not replace the need for quality assurance measures implemented throughout design, development, and maintenance."{initial comment: Sharron}

Introduction section

Not clear about the term "common functionality". Common to what? Could another term be used that denotes the completion of a task - "Task achievement" perhaps? Rationale: "Common" is not the right word for this and authors do not want to use "essential"{Sharron 28 March}

Step2b:Identify Common Functionality.I keep reading the definition of "common functionality) and I'm just not sure what it is we're talking about here. Are we talking about main navigation, header, footers and so on? Specific features that is used site wide? Rationale: Wouldn't it be helpful to refer to "common functionalities" as representative testing use cases for the evaluation of the website? {Denis 28 March}

Step2b: . As noted above, I believe "common" to be the wrong word here. Suggest authors think of a word more likely to convey the sense of ability to complete the task at hand.
Rationale: "Common" is just not the right word for what I think this is trying to mean.{Sharron 28 March}

Step2b: . Not simple. Brainstorming: "core functionality", "key functionality", "site functionality" or would "purpose and functionality" work (i.e., looking at what follows in the second paragraph)? {Vicki - 2 April}

Background Reading, linked headings: (minor issue) Unlink the headings and add those links in the lists. linked headings discussion - EO meeting minutes of 4/5Rationale: It's inconsistent and a bit cludgy for the some headings to be links and not others. Also, it would be best to properly indicate the link to the WCAG Overview. Also, people with some link indicators turned off (e.g., underlining) will miss that the headings are links. {initial comment: Shawn}

Introduction "need". Replace the words "need" and "needs". Probably replacing with "should" would be simplest; however, we understand there issues with that word.
See brainstorms in needs - EOWG minutes 5 April

initial comments

Don't like the repetition of "need," "needs to"
Rationale: It doesn't actually make sense, seem true. Better to use "should" {Sharron, 28/March}

Rationale: content could well be in the process of being written or content could already be available. In other words, content creation may not necessarily take place later on. {Vicki -31 March}

Abstract/Intro/Scope: - Tighten it up. It should be a short summary. Best if the whole this is not exactly the same wording as later in doc.

initial comments

A large part of the Abstract is repeated in the Introduction and Scope section. Is this really intentional?
Rationale: I think documents like these are already long enough, without some content being repeated. {Denis 29 March}I think it's common practice for the Abstract to be a summary of material from the main document, and therefore should repeat info.{Shawn}

Purpose of the Document: Re-write the list to be more clear, and to focus on the most common uses first. fyi, see Who WCAG-EM is for in Overview page.Rationale: As written, the repetition of words makes it quite difficult to parse. Also, we are concerned about having "website developers" as the first item. See: Shadi (for discussion in EOWG f2f before CSUN).
(Below are a couple ideas from individuals. EOWG didn't have time to work through these.)

initial comments:

The claim that WCAG-EM is for web developers who are interested in accessibility, will probably discourage more developers than it will encourage. WCAG-EM is for developers who specialize in accessibility or who want to specialize in accessibility. It is way to exhaustive for a general web developer who wants to write accessible code. I fear that casting the net too broadly will discourage many well meaning developers, and may inhibit adoption by people who need it. Possible Danger: Administrators who are ignorant of the divisions of labor in a web development project may assign this document to all developers, and accidentally cause push back from developers who do not need and cannot tolerate this level of detail. {Wayne}

The readability of the "Purpose of this Document" section seems difficult. Because the bullet items are long, they seem too bunched together at their current spacing and too wide for easy reading of a list. Also, many of the lines begin with very similar wording: "web accessibility" begins 4 of the 8 lines; web or website begins every line. Lists with similar wording, especially at the beginning reduce readability since it's difficult to distinguish one line from another. Perhaps some lines could start with a distinct function instead of the type of user. Or set up the list as a table with the type of user on the left column and the function or purpose in the right column.
For example:

Trainers and educators who want to teach methods and approaches for web accessibility evaluation

Accessibility Benchmarking

Individuals who want to benchmark or compare the accessibility conformance of websites

I'm not sure I know the best solution, only that the readability of this section is problematic. {Howard, 21/March }

[further comment] Agree with the difficulty of reading. Very much like the table.{Sharron, 28/March }

The "who" refers back to a process, not a person, in the bullet item "Web accessibility monitoring activities who want to benchmark or compare the accessibility conformance of websites" under "Purpose of Document." One possible solution: change "monitoring activities" to "monitors."{Howard, 21/March }

Observation: The list indicates use cases that "want" to perform an activity, how about "need" to perform an activity? Possibly, by shortening the items for easier reading, and including some "need" to perform activities, the list might not seem repetitive (i.e. 4x starting with "Web accessibility...") and may be easier to read. (I would even remove the repeated instances in the bullet points "of websites" since it it's clear in the context of the document that the reference is to accessibility conformance "of websites"). Suggested rewording with shorter sentences:
• Accessibility evaluation service providers who need to evaluate websites for accessibility conformance
• Experts performing accessibility monitoring activities who need to benchmark or compare the accessibility conformance of websites
• Consultants who need to analyze and report on accessibility conformance of websites
• Developers who want to evaluate accessibility conformance of their websites to monitor or improve them
• Website owners, procurers, and suppliers who want to learn about the accessibility conformance of their websites
• Researchers and disability advocates who want to explore accessibility conformance practices
• Accessibility trainers and educators who want to teach methods and approaches for web accessibility evaluation
• Web masters, content authors, designers, and others who want to learn more about web accessibility and evaluation{Vicki - 1 April }

initial comment: the first para under 'Scope' would be easier to read as a list rather than a ';' separating parts of a long para {Andrew, 15/March}

[further comment: Scope: Yes, agree, presentation using a list style would make the scope more readable {Vicki - 31 March}

Purpose of this document, last sentence: Delete the sentence starting with "Complementary resources on ..." and add those resources to the next section "Background Reading".

initial comment: I agree with the readability difficulties. It could also be improved in the paragraph beginning with "Complementary resources on"
Suggestion: Begin with: For further information, see complementary resources below." Then list those resources in a bulleted list. {Sylvie 22/March}

Using this Methodology section

Using this methodology, first paragraph: Reword to be more positive and be in-line with Easy Checks (which will replace the old Preliminary Eval page). For example, something like:

This methodology is a thorough, robust approach for evaluating conformance to WCAG 2.0. Before applying this methodology, it is usually best to do a preliminary evaluation to identify obvious accessibility barriers and get a general idea of the level of accessibility of a website. A simple, quick approach for getting a general idea if a web page has accessibility barriers is provided in Easy Checks - A First Review of Web Accessibility.
{EOWG ran out of time to discuss this more. We can try again later...}

initial comments:

In second sentence: "A more cursory approach for exploring the accessibility of a website", the term "cursory approach" is not clear to me. Is it jargony? Why not using a term that everyone can better understand? {Sylvie 22/March}

For better readability, it may better to split following sentence in two:
Actual text: "In many cases it is advisable to carry out such preliminary reviews before applying this methodology, for example to identify obvious errors and to develop a rough understanding of the overall performance of the website."
Suggestion: "In many cases it is advisable to carry out such preliminary reviews before applying this methodology. For example, identify obvious errors and develop a rough understanding of the overall performance of the website. {Sylvie 22/March}

previous draft rewording: A preliminary review can be quite useful to identify obvious errors although it will not check every accessibility issue and will likely not catch all of the problems on a site. Nevertheless, to develop a rough understanding of the overall performance of the website, reviewers may conduct such an exploration using WAI's Easy Checks - A First Review of Web Accessibility
Rationale: Emphasizes usefulness rather than "cursory" {Sharron 28 March}

Confused between paragraph above "Example of a website diagram" and one below. Above paragraph seems to indicate that a self-contained subsection of a website would qualify for this evaluation as a "self-enclosed" site. In the paragraph below the diagram, the implication is that "sub-sections" of a university site may not be considered self-enclosed sites: "In the example above, none of the depicted parts may be excluded from the scope of evaluation in the context of this methodology, if it is to be applied to the university website." Perhaps this is referring to evaluating the entire university site as a whole. Anyway - it's a confusing section. I can't offer a rewording until I know for sure the exact meaning trying to be conveyed. (Howard 4 April)

Duplicate sentence: Review Teams and immediately following section, Involving Users both begin with the same sentence. "The methodology defined by this document can be carried out by an individual evaluator with the skills described in section Required Expertise." Reconsider if this sentence is really necessay in both places. If so, consider that it will look like an error to some people, so see if there's aother way to cover the information.

initial comment:

Is this really necessary?Rationale: I don't understand what it adds to the point of either of these sections. {Sharron 28 March}

Particular Types of Websites, Web Applications section: Rewrite to be more clear.
(Below are a couple ideas from individuals. EOWG didn't have time to work through these.)

initial comment: In second bullet "Web Applications", the note of this part is very long due to the many links and few sentences. Is it possible to make it clearer? After I read it twice, I still try to understand it. {Sylvie 22/March}

I can understand why the definition may be difficult to understand. Below is a suggested abridged version:Websites with a limited number of pages which dynamically generate content are considered as a single website (or part of a larger site). Each state of a web page and content-generated web page for the purpose of this document should be included as a candidate for sampling. {Vicki - 1 April}

Conformance Evaluation Procedure section

The paragraph describing the flow diagram has visual dependencies. Change to:
"The workflow diagram depicts each of the steps defined in this
section. Work flow starts with the first step and proceeds to the last
step in order. In addition, the diagram permits evaluators to return
to any preceding step in the process whenever the evaluation reveals a
need to rework a sequence of steps." Rationale: Describing arrows. The new description is a functional description of the picture. {Wayne}

Step 1: Define the Evaluation Scope Section

Rewrite this section to be more clear and positive. Hopefully can point to better guidance in the WCAG Techniques documents &/or elsewhere soon. Note that there are currently not many (any?) published techniques for meeting WCAG 2 other than WAI's, so that may be a source of confusion. However, lots of people have worked on evaluation criteria. Is this section intended to include eval criteria as well? If so, that should be made clear.

initial comments:

Step 1d "However, it is not necessary to use these particular techniques. In fact, in some situations, such as in a closed network, it may be necessary to use techniques that are specifically developed to the particular needs of the users of that network. Individuals and organizations developing techniques have to employ methods for establishing the technique's ability to meet the WCAG 2.0 Success Criteria." without support .
Rationale: Wouldn't it be useful to provide one or two examples of this? So it makes more sense to those reading this document? {Denis 28 March}

Step 1d In addition to providing examples, this should not be an optional part of the methodology. It seems critical for the reviewer to define what techniques are being used to measure conformance or nonconformance to the SC. The section could make note of the fact that reviewers are not bound by previously established W3C Techniques and Failures, but it should still require that some documentation is needed of just what Techniques are being used. Could also encourage the addition of those new Techniques to the existing library.
Rationale: Without techniques being defined, the review is in danger of becoming undocumented opinion rather than measurement of conformance to a standard. {Sharron 3 April}

Step 1a Sentence: "This scope definition may not contradict the terms established in section Scope of Applicability."
EOWG sees the need for the sentence. since some found it confusing, please try to make it more clear.

Step 2: Explore the Target Website Section

Step2d: Sentence: "Only the web technologies that are relied upon to provide the website need to be identified for evaluation."
Consider adding "functionality" after the word "website" or find another way to make it clear.

initial comment:

Comment: I feel a word is missing, can we add "functionality" after the word website so that it reads:"Only the web technologies that are relied upon to provide the website functionality need to be identified for evaluation." {Vicki - 2 April}+1 {Sharron - 3 April}

Step2d: . "Identify the technologies relied upon to provide the website."
Consider changing "provide" to "render", "produce", or something like that. Or changing it to "relied upon to provide the website functionality".

initial comment:

Would change "provide" in "Identify the technologies relied upon to provide the website." to "render" or "produce."
Rationale: You "provide" food or lodging but not a website. {Howard - 4 April}

Comment from Howard (above) ties in with the previous comment and we discussed on the 12 April telecon maybe using for both mentions "relied upon to provide the website functionality" {Vicki - 12 April}

Step 3: Select a Representative Sample Section

Section Overview Note Consider if this is needed here and later. Right now it's too much text and repetitive. Can it be only in one place? Can you edit it significantly?

initial comment:

What is the point of the Note? Since it is to be covered in a later section, it seems redundant to mention here.Rationale: Tersification. It is an entire unnecessary paragraph {Sharron March 28}

Step 4: Audit the Selected Sample Section

Step 4 Note 2 Make the first point stronger and more clear. Make the second more clear that it's optional. For example, maybe something like:
"If there is no content related to a success criteria (for example, no video), then the success criteria is "satisfied" or "met", according to WCAG 2.0. Optionally, an evaluation report can specifically indicate success criteria for which there is no relevant content, for example, with 'Not Applicable'."
(Also, does this belong here? or in reporting?)

initial comment:

Add text to say: "Note: According to WCAG 2.0, Success Criteria to which there is no matching content are deemed to have been satisfied. An outcome such as 'Not Applicable' may be used to denote the particular situation where Success Criteria were satisfied because no relevant content was applicable,
<new text>
but WCAG 2.0 does not require this. A Success Criteria that is either validated or not applicable, for whatever reason, is considered satisfied."
</end new text>
Rationale: To explain more completely {Denis 28 March}

Step 5: Report the Evaluation Findings Section

Step 5.b: Conformance Evaluation Statement (Optional) Change:
"As per the requirements for WCAG 2.0 conformance claims, also accessibility evaluation statements have to include the following information:"
To:
"As per the requirements for WCAG 2.0 conformance claims, accessibility evaluation statements also have to include the following information:".Rationale: Grammar {Dennis 28 March}

Step 5.b Note Delete the negative note at the end and add a positive statement at the beginning of the section saying taht conformance claims are for completed websites. Also, considef making "WCAG 2.0 conformance claims can only be made for the web pages that have been actually evaluated and identified to conform with its conformance requirements." more positive.

initial comments:

It is not possible to make an accessibility evaluation statement for a website that is still in development."
But would be improved by saying:
"An accessibility evaluation statement should not be made for a website that is still in development."Rationale: Again, WCAG-EM is not a dogma or a law. We shouldn't impose anything. Rather than prohibit making an accessibility evaluation statement for a website that is still under development, maybe it would be better to suggest not to. {Denis 28 March}

Copyedits & things that probably don't need discussion

In first sentence of second paragraph of Abstract, should be comma after "for example"{Howard}

Typo: (may be several times in document): technqiues instead of techniques. {Sylvie 22/March}

Consistency of section headings: The link text of sections (steps) that are referred to are not always the same as the sections titles. For example, at the end of section step 5.A, there is a link to "Step 5.b: Provide and Accessibility Statement)". First there need to be corrected in "an accessibility Statement". Second, the title of the heading is: "Step 5.b: Provide a Conformance Evaluation Statement (Optional)", whereas the Methodology Requirement 5.B is entitled: "Provide an accessibility conformance evaluation statement (Optional).". {Sylvie 22/March}

Define Scope Overview: Well done in explaining the relationship/role of the reviewer, the commissioner, and the owner.Rationale: Important distinction, well made up front.{Sharron 28 March}

Overall Comment: This is thorough. That is a good thing. Tom and I agreed that this would have been great for us when we started our giant accessibility project at the California State University. We had to discover so much of this on our own. The WCAG-EM group managed to pull together a lot of hard earned experience into a truly useful guideline for someone who needs to bring a site into complete Level 2 conformance. Overall, it reads well. EO has caught most of the bumps. Combined with the Easy Check from EO (for work-a-day) review. This is a powerful package. {Wayne Dick and Tom Jewett 4/10}

Comments outside EOWG Scope

The following comments were proposed by EOWG participants; however, EOWG did not discuss them because they were probably outside EOWG scope (and we have limited time and lots to do :-).

Step 3 Intro: Instead of creating pressure on orgs to consider assessment of ALL pages, why not focus on trying to get them to think about selecting the best possible samples within reasonable scope, rather than have them perceive this selection as a failure because they can't afford to cover all pages? Rationale: I know we mention it's usually not possible to evaluate all pages, but if we can recognize that it's rarely possible, then why say ideally it should be all pages? That creates unrealistic expectations and a pressure most people do not need to deal with. I feel this is wishful thinking - we need to be more realistic about this. Most accessibility evaluation do not cover all pages and no organization can ever afford to run a complete evaluation of all their pages, except for very small sites (and even then, if they have a small site, their accessibility budgets will tend to be proportionally small) {Denis 28 March}

Step 3e: Random Sample 5% seems a little low to me, even when the sample contains over 30 to 50 pages total (WCAG-EM implies it should always be more than this). I would suggest around 10-15% (where 50 pages would include another 5-7 random pages to complete the sample).{Denis 28 March}

Step3e: Random Sample. I actually disagree with Denis' email objection to the 5% sample. I think %5 is an adequate number of random pages to add to the mix.Rationale: If the reviewer has followed the advice and chosen the samples well, it is unlikely that the random sample will provide much (or any) value. But it is good to do as a process check.{Sharron 29 March}

Step 4 Note 1 We should suggest isolating template findings from the rest of the findings related to main content of a page or pages, so findings/issues related to templates can be referenced globally for all related pages, rather than repeating the findings every time.Rationale:By doing this, once the template findings are covered, all subsequent pages could only be covering the main content issues, making the whole process simpler and more organized. {Denis 28 March}

Evaluating a Large Website with Separate Parts: Would it be useful here to suggest personas? To think about the different uses that people will have for the large website and to define those pathways? For example, a government agency site may have people coming who are looking for general information, or they may be seeking specific services. There could be a robust series of pages that do not require log-ins as well as those that do. Interest/interaction may be regional and there is likely to be a different series of pages for agency staff. Rationale: I have found that if the commissioner of the review will define the stakeholder/constiuent groups who most frequently use the site and what they do there, it can help define critical pathways for various tasks. {Sharron 3 April}

Re-Running a Website Conformance Evaluation:
I would like to suggest a proportion of 60%/40% between portion of web pages that were in the original sample and additional new web pages that were not in the original sample.Rationale: to get significant distinction between old pages and new ones and measure how well they compare once remediation has been applied to webpages. {Denis 28 March}

other

fyi, Comments that individuals sent directly (not through EOWG)

Suzette Excess use of 'per' especially in section 5. I'm not sure how to do an archive link but have sent the following direct:
"I have read through WCAG EM and congratulate you on its progress. On a small point of language I noticed is an outbreak of the word 'per' - especially in section 5, which I would suggest could inhibit ease of understanding and is also unnecessary. I would propose removing all 'per' and using a plain language alternative, if necessary.
FYI:
I researched the opinions of others via Google, and here is one of the more
interesting discussions on the use of per and as per:
http://www.businesswritingblog.com/business_writing/2009/05/your-help-please-with-as-per.html
Per and 'as per' is a pseudo-latin construction. Opinions appear to divide between clarity, efficiency, and improper use. There are however alternatives which are more natural.
The only examples that fit well in my head is taking medicine 'per
day', or sharing pencils 'one per child' and 'per your instructions'. The use of 'Per website' and 'per web page' (WCAG EM methodological requirement 5c) seem particularly puzzling and drove me to Google to find out why. {Suzette}

Archived Notes (not for Eval TF)

[closed. updated Style with rationale for putting periods and commas outside of quoted terms.] This may need discussion, but I'm not sure where else to put it. According the Chicago Manual of Style Rule 6.8: "Periods and commas precede closing quotation marks, whether double or single." The way I did the previous sentence conforms to that. Things like "not "magnifying glass"." don't. We use this pattern a lot, but that means we're consistent. It just may be consistently wrong. OTOH, it would be an incorrect pattern in code, so coders may cringe if we adopt the Chicago rule. (3/20/13) {Anna Belle}

Comments submitted on 22 October 2012

Readability & Usability

What specific revisions will make the document more readable and usable?

linked terms distracting- Every time a term is mentioned, there's a link to the definition e.g. "web pages",websites, I find this quite distracting and I think it reduces the readability {Vicki}.
Strongly agree. Some ideas: just link it the first time in the document, or a new section. OR provide an option for users to turn off these links. {shawn}

linked references in sentences - the linked reference (e.g. "3.5 Step 5: Report the Evaluation Findings") in the middle of a sentence is also a little distracting and breaks the reading. {Vicki}

examples matching use cases - readability, section 1.1 scope and 1.2 audience. I think this is a difficult read for the intended audience. To improve usability I would suggest there is a need for better tailored use cases that match 1.2 and are then addressed consistently. There seem to be some examples included about commerce but I think it needs to use a better defined range - that better capture the context of the audience be it large scale government sites, large commercial sites, banks and utilities, and big agencies through to smaller operators eg a local school, a local charity, small organic food retailer. {Suzette}

Delete text from Abstract - Abstract current wording: "This work is part of complementary W3C/WAI activities on Web Accessibility Evaluation and Testing that address different W3C/WAI guidelines and specifications."Comment: This leads reads off the page to 2 separate places. Suggest not having it in the Abstract. OK in the intro, but in the abstract, I think it takes the focus off this document too much. {Shawn}

Section numbering - Please consider not numbering every section. "3.4 Step 4:" etc adds complexity. It would be much simpler to just have "Step 4:" {shawn}

[minor] delete "W3C/WAI" where you can

[minor] add "(WCAG-EM)" to the title, abstract, and introduction

Scope

Is it clear that this set of methodologies is only for websites after they are built and does not address methodologies that might be used prior to and during development? If not, please suggest ways to clarify the scope.

Given that the title is about Compliance evaluation, I have no problem with clarifying that the scope is what educationalists call summative assessment (like the final exam!) rather that formative assessment (feedback you can learn from). I noted a number of mentions in the opening sections about the benefits or effects of iterative testing during the design process.
My suggestion is to keep the two types of testing separate because they relate to substantially different scenarios. Developers probably should not do their own formal compliance testing. I would suggest that the EM has a short paragraph to discuss the difference so that the reader is aware of this and can chose whether the Compliance EM document is relevant or if they need to go to the less formal 'Design_and_ Evaluation' section. {Suzette}

"However, this methodology only addresses evaluation of website conformance to WCAG 2.0 after its development. Complementary resources from W3C/WAI on evaluation provide guidance on evaluation in other situations, and complementary W3C/WAI activities on Web Accessibility Evaluation and Testing address different W3C/WAI guidelines and specifications."

"Also, WCAG 2.0 conformance requirements apply to individual web pages. However, on most websites it is not feasible to evaluate every web page with reasonable effort. To ensure conformance of a website to WCAG 2.0 it is necessary to ensure that this is considered throughout the development process. Following this methodology helps assess the level of conformance with reasonable confidence where it is not feasible to evaluate every web page."

I prefer not to restrict the evaluation to "only" sites after development.

I find the second paragraph starting "Also, WCAG 2.0 conformance....etc." somewhat puzzling. In the first paragraph, there is mention that the evaluation methodology is only for sites after development but, on the other hand, in the second paragraph, there is mention of the importance of conformance during the development process and following "this methodology" (which in the first para is stated as only for after development) helps assess the conformance.

Suggestion (if the scope is fixed on already developed sies):

This methodology is best used to address evaluation of website conformance to WCAG 2.0 after development. WCAG 2.0 conformance requirements apply to individual web pages. On most websites, it can be challenging to evaluate every single web page. This methodology helps assess the level of conformance with reasonable confidence where it is not feasible to evaluate every web page.

Complementary resources from W3C/WAI on evaluation provide guidance on evaluation in other situations, and complementary W3C/WAI activities on Web Accessibility Evaluation and Testing address different W3C/WAI guidelines and specifications.

It is noted that conformance which is considered during the development process reduces significantly the costs of evaluation after development.
{Vicki}

[important] The only one comment I do feel somewhat concerned about is the "Scope" and restricting the document to "only" sites after development. I look at it from the point of view of someone e.g. a developer coming across the document through a search and, with the patience (rather lack of it) that we have on the web, it might be disregarded by the restrictive focus. It is also a document which could still be useful to someone who might perform iterative evaluation through the development process. Therefore, I would recommend being more receptive to a wider audience by changing the wording somewhat. {Vicki via Sharron}

Evaluate throughout

How should we communicate the importance of evaluating throughout website development, not just at the end?

AbstractAbstract current wording: (in a paragraph) However, ensuring and maintaining accessibility requires that accessibility is considered throughout the development process. Complementary resources from W3C/WAI on evaluation provide guidance for other situations.Abstract suggested text:(separate paragraph) When developing or redesigning a website, accessibility should be considered early and throughout the design and development process. This is addressed in a new resource under development{which EOWG expects to have drafted for the next WCAG-EM draft so you can put the actual title of the page here}.{Shawn}

Introduction, Scope, middle paragraph:Current text: "However, this methodology only addresses evaluation of website conformance to WCAG 2.0 after its development. Complementary resources from W3C/WAI on evaluation provide guidance on evaluation in other situations, and complementary W3C/WAI activities on Web Accessibility Evaluation and Testing address different W3C/WAI guidelines and specifications."Suggested text: This methodology only addresses evaluating an existing website's conformance to WCAG 2.0. When developing or redesigning a website, accessibility should be considered early and throughout the design and development process. This is addressed in a new resource under development{which EOWG expects to have drafted for the next WCAG-EM draft so you can put the actual title of the page here}.(new paragraph) Related information outside the scope of this document is provided in Evaluating Websites for Accessibility, W3C WAI Web Accessibility Evaluation and Testing Activities, and WAI Guidelines and Techniques. {shawn}

Missing?

Are there any major considerations that we have overlooked or misrepresented?

Updates, maintenance, drift: An accessibility audit is static - a snapshot in time. By contrast, contemporary websites are organic - constantly evolving and growing. New material and novel features can be added by people who were not part of the original development team and these authors may have very limited editorial skills, and knowledge of development. As part of EM, I would suggest that that we should make a strong case for planned reviews to maintain standards and avoid the effect of drift that has been observed by researchers. {Suzette}

Related material

How much should we refer to and link to existing material as reference and how much should we include directly in the new material? Balance the inconvenience of jumping back and forth between existing documents against our reluctance to unnecessarily duplicate information.How should we align the WCAG-EM work with the Preliminary Evaluation document now in process at EOWG?What should the Conformance Evaluation page be in relation to WCAG-EM? (see also Eval Analysis)

Other

These comments were entered before we had the structure above, so might apply to above or not.

[Very Important]Methodology Requirement 1.d - This requirement appears to claim that the website owner and evaluator are free to choose which users may be excluded. Human rights do not work that way. What reasonable base technologies are required for conformance? This is particularly disturbing within the context of intranets where employers may choose to limit users to assistive technologies the employer can purchase cheaply. The note at the end is not sufficient.
Example, many sites prescribe large print with high contrast to be a universal cure for low vision. This discounts the need for albinos to have lower luminosity. Without high luminosity, high contrast is not possible. {Wayne Dick}

[minor] Relied Upon - Just hangs there without inclusion in the context of accessibility supported. {Wayne Dick}

[Med] The note to (Web Applications): It may not be feasible to exhaustively examine all possible settings, but sampling should apply just as any website. {Wayne Dick}

3.1.5 Step 3. The evaluation commissioner, must be informed that if the self defined sufficient techniques are used then a future accessibility audit may ask for a proof of sufficiency. {Wayne Dick}

Throughout 3.1.5 PDF and Silver light are basic web technologies. They are no more technologie than docx or raw text. They as basic web technologies. The accessibility open question.
This issue is particularly important if this document is going to be normative. I do specific reference to Adobe or Microsoft products as basic web technologies is serious vendor preference. {Wayne Dick}

3.3 Step 3: Select a Representative Sample, (First sentence) "While ideally every web page of a web site is evaluated, usually this is not possible on most web sites."
I feel that stating "usually this is not possible on most web sites" sounds somewhat contradictory (to our objective).
Suggested text: "While ideally every web page of a website should be evaluated, this may not be possible for very large websites." {Vicki}

3.3 Step 3: Select a Representative Sample - website with many heterogeneous sub-university example, the sample selection include users. Focus groups are an effective probably necessary.
Large heterogenous sites behave more like a set of independent sites placed under one URL. As such it is important to organize focus groups within the different autonomous site owners to determine usage patterns and prioritize pages for evaluation. When organizing the CSU System, 23 Campuses, each having more than 50 academic departments, and more administrative departments, we found that the IT unit could not get a handle on usage patterns on its own. Individual site owners had to give direct input on priority and usage. Example: At CSU Long Beach the Computer Science lab invokes the Computer Science Department Website every time a student logs into a lab machine. That makes the Computer Science Department the most referenced site on the Campus. But it is not the most important. {Wayne Dick}

Responding to the term "with reasonable confidence": I have no problem with the term. Alternatively, if the above term is a point of discussion, you could simply modify the sentence to "that is representative of the entire target website". {Vicki}

2.5 Involving Users (Optional)- Second last sentence. "While not required, it is strongly recommended to involve real people covering a wide range of abilities...."
Suggest removing "real". {Vicki}

3.1.4 Methodology Requirement 1.d: Third paragraph: second last sentence. "For example, ….., then the website is effectively not accessible for some users."
Suggest adding something about conformance for emphasis e.g., "and therefore does not conform to the specified conformance level of WCAG 2.0. (3.1.3 Step 1.c.)" {Vicki}

Term "Common Functionality"
Feedback was requested on the definition on "Common Functionality".
Having read through the document, I don’t have a problem with the term "common functionality. However, it might pose a question mark to some. Therefore, I’ve put forth a few options below depending on what you mean and if you wish to be more specific:
If it is the general functionality of the site, you could simply leave out the word "common" or use "basic functionality"
If "common functionality" has a more technical connotation (referring to the style sheets, javascript, classes, i.e., the technical mandatory requirements for multiple device support and functionality), leave it as "common"
Another suggestion in the line of "common" and "key" is “core”.
Suggestion to modify the description (in line with the "Note"):
Functionality of a website includes tasks that users of a website perform in order to achieve a goal. {Vicki}

3.5.3 Step 5.c: Provide a Performance Score (optional) I think there are good reasons for offering a score in addition to or as an alternative to the absolute measure of pass/fail. I have read this section several times, and tried really hard to construct a formula to help me comprehend the differences but I am currently confused. Maybe my basic scenario is wrong, so perhaps a worked example would help? Eg what would the score show where some form elements are consistently incorrectly marked and are affecting 5 out of 10 pages tested. {Suzette}

Term "sub-site" : do you want to add the definition to the terms and definitions? {Vicki}

Section step 1.A, last paragraphis not clear to me. What is necessary there? May be it could be easier to understand with concrete examples. <quote>"The outcome of this step is an unambiguous definition for the target website, so that for each web page it can be determined whether it is within the scope of evaluation or not. Where possible, it is recommended to use formalizations such as regular expressions or listings of Universal Resource Identifiers that define the web pages
that are within the scope of evaluation (part of the target website)."</quote> {Sylvie}

Please reconsider the purpose of this list in the context of WCAG-EM. (Remember this section starts with "It is assumed that the reader of this document is familiar with the following related resources from W3C/WAI:") I can see that Essential Components of Web Accessibility and How People with Disabilities Use the Web are particularly important for evaluation — perhaps only list those 2 — and not under the heading "Accessible Web Design" but something more directly related to understanding evaluation in the bigger context.
(minor point: This section is not parallel with the others above it — although I don't think that's a big deal. minor edit:link change: Web Content Accessibility and Mobile Web)
{Shawn}

Background Reading. current wording: "It is assumed that the reader of this document is familiar with the following related resources from W3C/WAI:" Ideas for edited wording: "To effectively use this methodology, you should be familiar with the following related information." or "The information below related to WCAG 2.0 and evaluation is important background for using this methodology. "
{Shawn}

Reports could also indicate the impact of issues found and specifically identify any high impact issues that are easy to fix. This could include: Any accessibility issues that interfere with the user completing tasks, issues that affect navigation and orientation, issues that occur very frequently and issues that can cause any loss or harm to a user. A lot of the important but technical information could be put in an appendix (such as documentation of each outcome of the steps as per Step 5.a: Provide Documentation for Each Step).

to:

Reports could also indicate the impact of issues found and specifically identify any high impact issues that are easy to fix. This could include:

Any accessibility issues that interfere with the user completing tasks

issues that affect navigation and orientation

issues that occur very frequently

issues that can cause any loss or harm to a user.

A lot of the important but technical information could be put in an appendix (such as documentation of each outcome of the steps as per Step 5.a: Provide Documentation for Each Step).

{Paul}

Copyedits & things that probably don't need discussion

3.1.2 Step 1.b. In-Depth Analysis. End of last sentence is missing a fullstop. {Vicki}

3.2.3 "Note: This step is intended to identify groups of web page instances, including in web applications."Suggest drop the "in" as in "including web applications" or add "including those in web applications". {Vicki}

[ http://www.w3.org/TR/WCAG-EM/#step3 third paragraph ] Change "…and in many cases these web pages may have different design to others" to "…and in many cases these web pages may have different design than others" {Paul}

[ http://www.w3.org/TR/WCAG-EM/#step5b ] Change "As per the requirements for WCAG 2.0 conformance claims, also accessibility evaluation statements have to include the following information:" to "As per the requirements for WCAG 2.0 conformance claims, accessibility evaluation statements also have to include the following information:" (change word order for clarity){Paul}

[ http://www.w3.org/TR/WCAG-EM/#step5c paragraph 2 ] Change "The type of scoring system used has to be indicated along with..." to "The type of scoring system used has to be indicated, along with..." (added comma) {Paul}

[ http://www.w3.org/TR/WCAG-EM/#step5c ] "Currently the following scoring approaches are provided by this methodology:" The word "Currently" suggests other scoring approaches will be added to this document in the future. Does it make sense to remove it?{Paul}

[ http://www.w3.org/TR/WCAG-EM/#reports Appendix C, "For In-Depth…" bullet 1 ] Change "...the report identifies if it is met or not met and provide minimally one example..." to "the report identifies if it is met or not met and at a minimum provides..." {Paul}

[ http://www.w3.org/TR/WCAG-EM/#reports Appendix C, "For In-Depth…" bullet 3 ] Change "Provide suggestions for improving the overall accessibility of the website and if possible suggesting guidance for the future;" to "Provide suggestions for improving the overall accessibility of the website, and if possible, suggest guidance for the future;" {Paul}

[ http://www.w3.org/TR/WCAG-EM/#terms Terms and Definition ] Headings that have anchors leading to them end up underlined, or behave differently in some browsers (they turn red when clicked in Safari, underlined in others), because they are coded like so: <h3><a id="terms" name="terms">Terms and Definitions</a></h3>. Either modify how it's coded to <h3><a id="terms" name="terms"><!-- anchor --></a>Terms and Definitions</h3>, or change the stylesheet to make sure they remain styled like headings only. {Denis}