WebAIM - Web Accessibility In Mind

E-mail List Archives

Re: What we found when we tested tools on the world's least-accessible webpage

for

From: Birkir R. Gunnarsson
Date: Feb 27, 2017 12:45PM

Mehmet

I really like what you are trying to do here, it is much needed and I
applaud the effort.
I will go in as soon as I have time and do more analysis for more
useful and specific feedback.

But there are several big picture factors I would think about and are
important for the accessibility tool selection:

1. Is a user problem you are noticing a WCAG violation. WCAG is far
from perfect and it does not cover all problems users run into.
Feedback on real issues encountered by real users with disabilities
and that are not covered by WCAG (2.0 A or AA) would be great for the
WCAG working group. But the accessibility testing tools are mostly
limited to actual WCAG violations to remain consistent (tools could
have optional usability warnings, but those should be clearly flagged
as such). Make sure that every violation you create is definitely a
WCAG vilation, and map it to the relevant success criterion (this wil
help further analysis, such as when individual tools fail, or where
tools fail in general).

2. Come up with some type of measurement on the amount of "false
positives" or non-WCAG violations reported by a tool, even something
assimple as the difference between total issues found and the issues
you found that the tool caught. This is important, because a tool
could report every potential or possible thing as an accessibility
violation. Sure, it could uncover the most issues, but it would also
create a huge amount of unnecessary work for developers and would not
be productive use of people's time (and,just for the record, I am not
accusing Tenon of this, it is a great tool written by one of the
leading experts in the accessibility sector). You need to deduct
points from tools that report false positives. But there is a tool,
not on your list, where you have to contact the vendor specifically to
request that it does not flag all AAA issues as errors. That shows
fundamental misunderstanding of WCAG.

3. Ensure all your tools test the DOM. I think it is almost definite
for all tools currently used, but this is important. You need to fix a
couple of your accessibility defects using JavaScript that runs when
the page loads (e.g. adding alt text to a couple of images or a label
association between a label and a form field).

4. Does the tool evaluate ARIA? ARIA has made accessibility testing
much more complex, because it can be used to add functionality and fix
stuff. When properly used (along with proper functionality), ARIA is a
valid solution and cannot be shrugged of by reporting its use as an
error.
The tool should check whether valid roles and attribues are used, and
if they are appropriate for the widget or element.

I know this is a huge undertaking, but these could be good
consideration for a future evolution of what you are trying to do
here.
You are off to a very cool start.

Cheers
-B















On 2/27/17, Jared Smith < <EMAIL REMOVED> > wrote:
> Mehmet -
>
> Thank you for the follow-up, and for the updates. This really is a
> very useful chart. I still worry a bit about the perception that
> identification of more issues is better for accessibility
> implementers, but it is informative to know what various tools test.
> And this has helped us identify a few items for implementation or
> improvement in WAVE.
>
> I would ask that you also update the "How did each tool do?" chart at
> https://alphagov.github.io/accessibility-tool-audit/index.html based
> on the updated results data.
>
> Thanks,
>
> Jared
> > > > >


--
Work hard. Have fun. Make history.