WordPress needs automated accessibility testing

The Make WordPress Accessibility Team needs help wrapping its arms around WordPress Core as new features get created while helping refine the existing codebase to make it more accessible to people with disabilities. Our small team can’t be everywhere and test everything, but we’re working to gather an accurate picture of what needs our help the most.

Once we have a better idea of what needs our attention the most, how do we maintain a good grasp of it? Automated accessibility testing can help in a big way here, and I’d like to start working to incorporate it into Core. I brought this up in our last Accessibility Team chat.

What’s automated accessibility testing? Similar to unit testing, this would allow developers to run a set of tests during their local development workflow of patches for WordPress via a command line tool. These tests would output results that would alert developers to potential errors and help them fix things like missing alt attributes or unassociated form fields and labels. Automated accessibility testing isn’t perfect and it won’t help us overcome all of our challenges, but it can help us find simple, easy-to-fix errors, alert us to large trouble spots within the codebase that need more manual testing and allow us cover a bit more ground. It’s a nice compliment to the manual testing and education we’ve done so far.

What do we need to consider when selecting a tool?

Flexibility and Freedom. These tools exist for one reason only: to work for you. Every single second you spend learning the UI, figuring out how to use it, or sifting through results is a second of development time that could go into actually improving the system. The tool must be flexible enough to offer the freedom to use it in any environment by any team of any size in any location.

Accurate. Developers of accessibility tools want to find every single issue they possibly can. This unfortunately often includes things that are, in context, irrelevant or of low impact. They also may require an additional amount of human effort to verify. This is a disservice. Tool vendors need to understand that their clients may not have the time, budget, or knowledge to sift through false positives. The user should be able to immediately eliminate or ignore anything that isn’t an undeniable issue.

Understandable The user of the tool needs to know exactly what the problem is, how much impact the problem has, where it is, and how to fix it. Overly general, overly concise, or esoteric result descriptions are unhelpful. If a tool can find a problem, then it can make intelligently informed suggestions for remediation

I think these are good goals to try to hit. Ideally, we want something that will have little to no impact on a developer’s workflow, allow us to select different test criteria and deliver accurate results we can act on.

As research, I’ve spoken to Jesse Beach. She’s a Senior Front End Engineer at Acquia Inc., a Drupal contributor, a contributor to QuailJS (a tool Drupal uses for automated accessibility testing) and a member of the Drupal Accessibility group. She’ll help us look at QuailJS as a possible automated accessibility testing approach. And hopefully, we can share accessibility knowledge between these two awesome open source projects! We talked about what we (WordPress Accessibility Team) want to accomplish with automated testing, how a tool might fit into a WordPress developer’s workflow, the future of QuailJS and a bit about accessibility in our different open source projects.

One other possibility on the short list of tools besides QuailJS includes Pa11y. We will likely look at others as well.

I like the idea of expanding our automated tests. For each of the possibilities (QuailJS, Pa11y possibly others), I think it would be good to lay out the requirements, benifits and costs.

Personally, I would like to see whatever tool we use fit into our existing testing flow. This means that it could be run from the command line so that we can run the tests on travisCI and as a part of the grunt precommit command.

Integrating into the existing testing flow is a high priority, probably only below selecting a tool that has accurate, configurable tests.

Do we have guidance on selecting or integrating with tools that may not be fully open source. For instance, maybe the tests we write/modify are all open source, but we hit a third-party api to actually get results that isn’t open source.

Selecting something that’s fully open source/GPLV2 compatible has preference of course, but I thought the question is worth asking.

[…] 508 accessibility standards. Accessibility team member David A. Kennedy wrote a great post, “WordPress needs automated accessibility testing” in which he mentions Quail and another tool, pa11y, and I recommend reading what he has to […]

I’m interested in accessibility, WordPress and testing. I’m a follower of @WPaccessibility and I’ve read the last “Accessibility Team Updates” on the blog. Here are a few information you might find helpful for the topic.

You could find some help with Tanaguru, an opensource accessibility checker (disclaimer: I’m its creator). Tanaguru offers different kinds of audit like page and site-wide audit, but for the expressed need to test the backoffice of WordPress with ATAG, you could use the “scenario audit”. For short, you record a scenario a user could follow (e.g. creating a post, adding an image, modifying the alt, publishing), and send it to Tanaguru, that in turn replays it. Scenario can be replayed when you want. (Technically speaking, it is based on bricks of Selenium.)

For the needs of automation, you can also plug it with continuous integration tools. We are actually playing with our own Jenkins (that we use for building Tanaguru).