Gian Sampson-Wild’s A List Apart article provides an excellent overview of WCAG 2.0 testability issues. Like Joe Clark’s article and many others before, it’s difficult to view an article as entirely objective when the author is clearly carrying a grudge against the WCAG working group. Despite my inherent suspicions, Gian provides several strong arguments for removing or modifying testability in WCAG 2.0. To summarize, Gian argues that the WCAG working group’s requirement that all success criteria be testable has resulted in less-than-optimal guidelines.
Benefits of testability
Limiting the guidelines to only those that are measurable and testable has some advantages. It allows it to be adopted or adapted in broader realms, particularly in legal arenas where it could arguably have the broadest impact. In many cases, it makes determining conformance much easier. It also allows conformance to be better determined remotely and in a more (though not at all entirely) automated manner.
Testability problems
On the other hand, Gian argues that because of the testability requirement, many accessibility techniques are not in WCAG 2.0, despite the fact that they would increase accessibility. Removing the testability constraint would allow recommendations for increasing access for those with some cognitive disabilities – these recommendations would almost certainly be untestable.
She also argues that many of the existing guidelines and success criteria are not very testable as is. I agree. The first and perhaps broadest success criteria (1.1.1) requires text alternatives that present “equivalent information”. But what is “equivalent information” and how would you ever test this? WCAG 2.0 defines something as testable if a machine can clearly answer “yes” or “no” to the test or if “at least 80% of knowledgeable human evaluators would agree on the conclusion.” 80% seems to be a totally arbitrary percentage. And how do you even define if a conclusion is agreed upon. There is essentially no way of testing whether the human testability definition has been met.
WCAG 2.0 also introduces some success criteria that contain seemingly capricious levels of testability. For instance, readability at a “lower secondary level” has nothing to do with the actual audience and seems to be an arbitrary measurement. Still, it is supposedly testable – though I’m not sure how. Determining if language is appropriate for a site’s content is no less measurable by humans than determining if alternative text provides “equivalent information”. The introduction of such testability levels disallows the ability for developers to generate content that best fits their unique audience, but instead it provides an arbitrary measure to which all content and presumably all users must prescribe.
Is human testability even possible?
Much (or perhaps most) of accessibility is subject to interpretation and that interpretation will vary greatly. Regarding alternative text, I challenge you to find ANY image that presents content from ANY web site and then get 8 out of 10 accessibility experts to agree on what the alt text should be. Beyond that, I think I can safely guarantee that 8 of 10 of the WCAG working group members wouldn’t agree on the alt text for the W3C web site logo. Taking the testability requirement quite literally would result in the vast majority of accessible pages not reaching even Level A conformance because you could not prove that 80% of evaluators would agree on all subjective aspects. Perhaps more important is that “human evaluators” shouldn’t be determining appropriate alternative text anyways – content creators should.
Is there a solution?
So we have a dilemma regarding testability. If WCAG 2.0 sticks to its testability mandate and keep its slightly limited and complex success criteria, it risks alienating itself due to an inability for developers to prove testability at the 80% level. Alternatively, it can allow non-testable, pseudo-testable, and more far-reaching recommendations to be included and then risk criticism and lack of adoption because it is not testable. Lack of testability was, after all, one of the primary complaints regarding WCAG 1.0.
Throwing out all testability would be a grave mistake. However, the working group would be greatly benefited by taking another look at the 80% agreement level for proving testability and also revisiting their mandate that all WCAG 2.0 success criteria be testable.
We aren’t trying for “objective,” a goal that only journalistic ingenues even bring up. We try for fairness and accuracy, and if we can’t manage the former all the time, we at least declare our biases.
Now: Demonstrate that the biases you see in my and Gian’s articles altered the facts of the articles.
I wish you would get over this routine you have in which criticism of the WCAG Working Group is a symptom of a deficiency in the complainer rather than the Working Group. If WCAG were a happy ship, there would have been fewer drownings.
Joe-
I never suggested that your biases alter the facts. Yet one can’t help but approach those facts (or more accurately, your interpretation of the facts) with caution when the expression of those biases borders on vitriol.
I’m familiar with the significant problems with the WCAG Working Group. Politics run deep. General trust in them and the W3C process is at an all-time low. But this article isn’t about the working group, it’s about the standards and a hope that the community can motivate them to salvage those standards despite what the WG’s track record might be.
How’s retirement? 😉
Just a comment re: the Joe/Gian ‘grudge’ angle…
It does appear to those outside that some people do have an axe to grind with WCAG., which makes you wonder whether it is always getting a fair hearing. On the other hand, the fact that there are a number of people who seem to have an axe to grind with the working group is also somewhat suggestive that there are significant problems in the way it works. I think both are fair comments to make.
As regards ‘testability’ on the whole, I think it’s much easier to explain something to people who aren’t accessibility advocates already if you can explain to them how to test it, so I’d hate to see it dropped entirely. I would however be happy for some critical guidelines, particularly in areas currently under-represented such as cognitive – to be included even if they weren’t easily testable.
I think that this article has a misunderstanding of human testability. I would agree that assistive technolgy experts might choose different alt texts. But I don’t think that is what is at issue here. The important thing is would they look at the others and agree “yes, even though this may be different from what I would write it is still an equivalent”. And I think that is a very different question.
It’s no diffferent from sighted people looking at the original picture. They might see a landscape and say there is a village beside the river. Someone else might say its a river with houses beside it etc… but they are all describing the picture.
Why not listen to the people with the disability instead of a bunch of sighted people making the rules all the time? I’m legally blind and half the crap you guys come up with makes a site harder to use Vs easier. Web Accessibility experts, indeed.
I am who I am –
I’m not sure if you’re referring to me, WebAIM, or WCAG, but either way we would certainly invite you to provide feedback or recommendations. Simply criticizing without providing any substance provides little benefit.
I’m not referring to you, nor to WebAIM but do refer to WCAG.
Why force sites to use accesskeys? Many of the accesskeys that are assigned cause conflicts. If accesskeys are asisgned in form controls and use also in links (the same keys) the ones for the links end up not working because the form control accesskeys take priority.
It’s very hard to access the “help” link on a page with an order form when the “help” link uses the same accesskey as a form control.
Yet, to pass AAA it’s a requirement that accesskeys have to be used.
There are a lot of things that WCAG does that make things more difficult for blind/low vision people, than it does to make it easier for us.
Why don’t they just work with some actual REAL blind people, or REAL people with low vision problems, and write rules from that perspective?
A person with perfect sight has no idea or understanding of what it’s like not to be able to see. They just use a screen reader to test it with, and say “Yep. This works.” Maybe for them — it does because THEY CAN SEE.
Honestly, the best approach is to work with people that actually have the disability. You sighted people can only do so much pretending, and then, it’s not too accurate.
Where is this listed in WCAG 2.0? I don’t find it anywhere. The only place I can see access keys mentioned is as one of many possible solutions for bypassing blocks of content.
There are several blind individuals and people with various other disabilities working on WCAG 2.0. Admittedly, I think their participation isn’t what it should be. And again, if you have specific comments or recommendations, I’m sure they’d be happy to hear your thoughts.
Indeed, someone who is not blind cannot know what it’s like to not see. This doesn’t, however, mean that they can’t understand the issues of screen reader access and write guidelines accordingly. I would never suggest that you couldn’t possibly write guidelines regarding blindness because you can’t see. It’s a rather illogical statement.
If “sighted people” are totally off base on everything we are trying to do to improve accessibility, as you seem to be suggesting, please educate us otherwise.
I didn’t say AAA requirement is in 2.0. If I have read correctly, 2.0 is just a draft and not the current standard, no? So therefore one would logically assume the reference to AAA is with regard to 1.0 current standard where accesskeys are indeed required.
As I said, I’m in the group of people that by legal definition, are considered blind. Though I can see some color, and blurred shapes — I am by no means a “sighted” person.
The point I am trying to get across is that I support your statements, that testability isn’t possible with regard to an automated system. You (used in the collective sense) fail when you remove the “human-ness” aspect from any system that is designed to test its accessiblity for humans.
For this, it’s important to rely more heavily on humans and test the system in a real world situation. Have the new guidelines (or even the old ones) tested by people that are blind or have low vision problems. Then you know for sure how much help and/or hinderance a rule creates?
Perkins has a “school for the blind” and I’d bet there’d be plenty of students willing to test the guidelines for a real world situation because many of the students do research online.
I personally know some other people (legally blind) that do web programming, and even web design.
We may be blind, but we are able.
I don’t own a website so I just listed the url of my favorite search engine there. I hope that’s OK?
Now that WCAG 2.0 is finalized, I wanted to add here that from my reading accesskeys are no longer part of the latest standard. There’s some interesting work that has been done about user definable access keys by Juicy Studio and the Musing blog.