Important note: This Wiki page is edited by participants of the EOWG. It does not necessarily represent consensus and it may have incorrect information or information that is not supported by other Working Group participants, WAI, or W3C. It may also have some very useful information.

<li style="padding-bottom:1em">the first para under 'Scope' would be aesier to read as a list rather than a ';' separate long para <span style="color: #808080;">{Andrew, 15/March}</span></li>

+

<li style="padding-bottom:1em">the first para under 'Scope' would be aesier to read as a list rather than a ';' separating parts of a long para <span style="color: #808080;">{Andrew, 15/March}</span></li>

Reminder: EOWG will focus on the types of questions below under #Comments submitted on 22 October 2012. If you have comments on specific technical points in the document, you can submit them directly yourself -- that is, e-mail them to public-wai-evaltf@w3.org 22 March 2013.If you're not sure, feel free to put your comments here and the Group will decide if they are ones you should send yourself.

March 2013 Comments on 26 February 2013 Working Draft

comment {name}

Introduction section

comment {name}

the first para under 'Scope' would be aesier to read as a list rather than a ';' separating parts of a long para {Andrew, 15/March}

Using this Methodology section

comment {name}

Conformance Evaluation Procedure section

comment {name}

Considerations for Particular Situations section

comment {name}

Application of this Methodology section

comment {name}

other

Copyedits & things that probably don't need discussion

comment {name}

fyi, Comments that individuals sent directly (not through EOWG)

link to e-mail in archive {name}

Comments submitted on 22 October 2012

Readability & Usability

What specific revisions will make the document more readable and usable?

linked terms distracting- Every time a term is mentioned, there's a link to the definition e.g. "web pages",websites, I find this quite distracting and I think it reduces the readability {Vicki}.
Strongly agree. Some ideas: just link it the first time in the document, or a new section. OR provide an option for users to turn off these links. {shawn}

linked references in sentences - the linked reference (e.g. "3.5 Step 5: Report the Evaluation Findings") in the middle of a sentence is also a little distracting and breaks the reading. {Vicki}

examples matching use cases - readability, section 1.1 scope and 1.2 audience. I think this is a difficult read for the intended audience. To improve usability I would suggest there is a need for better tailored use cases that match 1.2 and are then addressed consistently. There seem to be some examples included about commerce but I think it needs to use a better defined range - that better capture the context of the audience be it large scale government sites, large commercial sites, banks and utilities, and big agencies through to smaller operators eg a local school, a local charity, small organic food retailer. {Suzette}

Delete text from Abstract - Abstract current wording: "This work is part of complementary W3C/WAI activities on Web Accessibility Evaluation and Testing that address different W3C/WAI guidelines and specifications."Comment: This leads reads off the page to 2 separate places. Suggest not having it in the Abstract. OK in the intro, but in the abstract, I think it takes the focus off this document too much. {Shawn}

Section numbering - Please consider not numbering every section. "3.4 Step 4:" etc adds complexity. It would be much simpler to just have "Step 4:" {shawn}

[minor] delete "W3C/WAI" where you can

[minor] add "(WCAG-EM)" to the title, abstract, and introduction

Scope

Is it clear that this set of methodologies is only for websites after they are built and does not address methodologies that might be used prior to and during development? If not, please suggest ways to clarify the scope.

Given that the title is about Compliance evaluation, I have no problem with clarifying that the scope is what educationalists call summative assessment (like the final exam!) rather that formative assessment (feedback you can learn from). I noted a number of mentions in the opening sections about the benefits or effects of iterative testing during the design process.
My suggestion is to keep the two types of testing separate because they relate to substantially different scenarios. Developers probably should not do their own formal compliance testing. I would suggest that the EM has a short paragraph to discuss the difference so that the reader is aware of this and can chose whether the Compliance EM document is relevant or if they need to go to the less formal 'Design_and_ Evaluation' section. {Suzette}

"However, this methodology only addresses evaluation of website conformance to WCAG 2.0 after its development. Complementary resources from W3C/WAI on evaluation provide guidance on evaluation in other situations, and complementary W3C/WAI activities on Web Accessibility Evaluation and Testing address different W3C/WAI guidelines and specifications."

"Also, WCAG 2.0 conformance requirements apply to individual web pages. However, on most websites it is not feasible to evaluate every web page with reasonable effort. To ensure conformance of a website to WCAG 2.0 it is necessary to ensure that this is considered throughout the development process. Following this methodology helps assess the level of conformance with reasonable confidence where it is not feasible to evaluate every web page."

I prefer not to restrict the evaluation to "only" sites after development.

I find the second paragraph starting "Also, WCAG 2.0 conformance....etc." somewhat puzzling. In the first paragraph, there is mention that the evaluation methodology is only for sites after development but, on the other hand, in the second paragraph, there is mention of the importance of conformance during the development process and following "this methodology" (which in the first para is stated as only for after development) helps assess the conformance.

Suggestion (if the scope is fixed on already developed sies):

This methodology is best used to address evaluation of website conformance to WCAG 2.0 after development. WCAG 2.0 conformance requirements apply to individual web pages. On most websites, it can be challenging to evaluate every single web page. This methodology helps assess the level of conformance with reasonable confidence where it is not feasible to evaluate every web page.

Complementary resources from W3C/WAI on evaluation provide guidance on evaluation in other situations, and complementary W3C/WAI activities on Web Accessibility Evaluation and Testing address different W3C/WAI guidelines and specifications.

It is noted that conformance which is considered during the development process reduces significantly the costs of evaluation after development.
{Vicki}

[important] The only one comment I do feel somewhat concerned about is the "Scope" and restricting the document to "only" sites after development. I look at it from the point of view of someone e.g. a developer coming across the document through a search and, with the patience (rather lack of it) that we have on the web, it might be disregarded by the restrictive focus. It is also a document which could still be useful to someone who might perform iterative evaluation through the development process. Therefore, I would recommend being more receptive to a wider audience by changing the wording somewhat. {Vicki via Sharron}

Evaluate throughout

How should we communicate the importance of evaluating throughout website development, not just at the end?

AbstractAbstract current wording: (in a paragraph) However, ensuring and maintaining accessibility requires that accessibility is considered throughout the development process. Complementary resources from W3C/WAI on evaluation provide guidance for other situations.Abstract suggested text:(separate paragraph) When developing or redesigning a website, accessibility should be considered early and throughout the design and development process. This is addressed in a new resource under development{which EOWG expects to have drafted for the next WCAG-EM draft so you can put the actual title of the page here}.{Shawn}

Introduction, Scope, middle paragraph:Current text: "However, this methodology only addresses evaluation of website conformance to WCAG 2.0 after its development. Complementary resources from W3C/WAI on evaluation provide guidance on evaluation in other situations, and complementary W3C/WAI activities on Web Accessibility Evaluation and Testing address different W3C/WAI guidelines and specifications."Suggested text: This methodology only addresses evaluating an existing website's conformance to WCAG 2.0. When developing or redesigning a website, accessibility should be considered early and throughout the design and development process. This is addressed in a new resource under development{which EOWG expects to have drafted for the next WCAG-EM draft so you can put the actual title of the page here}.(new paragraph) Related information outside the scope of this document is provided in Evaluating Websites for Accessibility, W3C WAI Web Accessibility Evaluation and Testing Activities, and WAI Guidelines and Techniques. {shawn}

Missing?

Are there any major considerations that we have overlooked or misrepresented?

Updates, maintenance, drift: An accessibility audit is static - a snapshot in time. By contrast, contemporary websites are organic - constantly evolving and growing. New material and novel features can be added by people who were not part of the original development team and these authors may have very limited editorial skills, and knowledge of development. As part of EM, I would suggest that that we should make a strong case for planned reviews to maintain standards and avoid the effect of drift that has been observed by researchers. {Suzette}

Related material

How much should we refer to and link to existing material as reference and how much should we include directly in the new material? Balance the inconvenience of jumping back and forth between existing documents against our reluctance to unnecessarily duplicate information.How should we align the WCAG-EM work with the Preliminary Evaluation document now in process at EOWG?What should the Conformance Evaluation page be in relation to WCAG-EM? (see also Eval Analysis)

Other

These comments were entered before we had the structure above, so might apply to above or not.

[Very Important]Methodology Requirement 1.d - This requirement appears to claim that the website owner and evaluator are free to choose which users may be excluded. Human rights do not work that way. What reasonable base technologies are required for conformance? This is particularly disturbing within the context of intranets where employers may choose to limit users to assistive technologies the employer can purchase cheaply. The note at the end is not sufficient.
Example, many sites prescribe large print with high contrast to be a universal cure for low vision. This discounts the need for albinos to have lower luminosity. Without high luminosity, high contrast is not possible. {Wayne Dick}

[minor] Relied Upon - Just hangs there without inclusion in the context of accessibility supported. {Wayne Dick}

[Med] The note to (Web Applications): It may not be feasible to exhaustively examine all possible settings, but sampling should apply just as any website. {Wayne Dick}

3.1.5 Step 3. The evaluation commissioner, must be informed that if the self defined sufficient techniques are used then a future accessibility audit may ask for a proof of sufficiency. {Wayne Dick}

Throughout 3.1.5 PDF and Silver light are basic web technologies. They are no more technologie than docx or raw text. They as basic web technologies. The accessibility open question.
This issue is particularly important if this document is going to be normative. I do specific reference to Adobe or Microsoft products as basic web technologies is serious vendor preference. {Wayne Dick}

3.3 Step 3: Select a Representative Sample, (First sentence) "While ideally every web page of a web site is evaluated, usually this is not possible on most web sites."
I feel that stating "usually this is not possible on most web sites" sounds somewhat contradictory (to our objective).
Suggested text: "While ideally every web page of a website should be evaluated, this may not be possible for very large websites." {Vicki}

3.3 Step 3: Select a Representative Sample - website with many heterogeneous sub-university example, the sample selection include users. Focus groups are an effective probably necessary.
Large heterogenous sites behave more like a set of independent sites placed under one URL. As such it is important to organize focus groups within the different autonomous site owners to determine usage patterns and prioritize pages for evaluation. When organizing the CSU System, 23 Campuses, each having more than 50 academic departments, and more administrative departments, we found that the IT unit could not get a handle on usage patterns on its own. Individual site owners had to give direct input on priority and usage. Example: At CSU Long Beach the Computer Science lab invokes the Computer Science Department Website every time a student logs into a lab machine. That makes the Computer Science Department the most referenced site on the Campus. But it is not the most important. {Wayne Dick}

Responding to the term "with reasonable confidence": I have no problem with the term. Alternatively, if the above term is a point of discussion, you could simply modify the sentence to "that is representative of the entire target website". {Vicki}

2.5 Involving Users (Optional)- Second last sentence. "While not required, it is strongly recommended to involve real people covering a wide range of abilities...."
Suggest removing "real". {Vicki}

3.1.4 Methodology Requirement 1.d: Third paragraph: second last sentence. "For example, ….., then the website is effectively not accessible for some users."
Suggest adding something about conformance for emphasis e.g., "and therefore does not conform to the specified conformance level of WCAG 2.0. (3.1.3 Step 1.c.)" {Vicki}

Term "Common Functionality"
Feedback was requested on the definition on "Common Functionality".
Having read through the document, I don’t have a problem with the term "common functionality. However, it might pose a question mark to some. Therefore, I’ve put forth a few options below depending on what you mean and if you wish to be more specific:
If it is the general functionality of the site, you could simply leave out the word "common" or use "basic functionality"
If "common functionality" has a more technical connotation (referring to the style sheets, javascript, classes, i.e., the technical mandatory requirements for multiple device support and functionality), leave it as "common"
Another suggestion in the line of "common" and "key" is “core”.
Suggestion to modify the description (in line with the "Note"):
Functionality of a website includes tasks that users of a website perform in order to achieve a goal. {Vicki}

3.5.3 Step 5.c: Provide a Performance Score (optional) I think there are good reasons for offering a score in addition to or as an alternative to the absolute measure of pass/fail. I have read this section several times, and tried really hard to construct a formula to help me comprehend the differences but I am currently confused. Maybe my basic scenario is wrong, so perhaps a worked example would help? Eg what would the score show where some form elements are consistently incorrectly marked and are affecting 5 out of 10 pages tested. {Suzette}

Term "sub-site" : do you want to add the definition to the terms and definitions? {Vicki}

Section step 1.A, last paragraphis not clear to me. What is necessary there? May be it could be easier to understand with concrete examples. <quote>"The outcome of this step is an unambiguous definition for the target website, so that for each web page it can be determined whether it is within the scope of evaluation or not. Where possible, it is recommended to use formalizations such as regular expressions or listings of Universal Resource Identifiers that define the web pages
that are within the scope of evaluation (part of the target website)."</quote> {Sylvie}

Please reconsider the purpose of this list in the context of WCAG-EM. (Remember this section starts with "It is assumed that the reader of this document is familiar with the following related resources from W3C/WAI:") I can see that Essential Components of Web Accessibility and How People with Disabilities Use the Web are particularly important for evaluation — perhaps only list those 2 — and not under the heading "Accessible Web Design" but something more directly related to understanding evaluation in the bigger context.
(minor point: This section is not parallel with the others above it — although I don't think that's a big deal. minor edit:link change: Web Content Accessibility and Mobile Web)
{Shawn}

Background Reading. current wording: "It is assumed that the reader of this document is familiar with the following related resources from W3C/WAI:" Ideas for edited wording: "To effectively use this methodology, you should be familiar with the following related information." or "The information below related to WCAG 2.0 and evaluation is important background for using this methodology. "
{Shawn}

Copyedits & things that probably don't need discussion

3.1.2 Step 1.b. In-Depth Analysis. End of last sentence is missing a fullstop. {Vicki}

3.2.3 "Note: This step is intended to identify groups of web page instances, including in web applications."Suggest drop the "in" as in "including web applications" or add "including those in web applications". {Vicki}