The Planning, Evaluation, Repair and Maintenance Process
The Importance of Human Evaluation

Overview

The two basic approaches to accessibility evaluation are:

1. Use a software tool
2. Use a human evaluator

Usually the best approach is to use both a software tool and a human evaluator. Each approach has strengths and weaknesses which compliment the others and form a more complete approach to web accessibility evaluation. People with disabilities can be especially valuable as accessibility evaluators, though this approach also has weaknesses.

The Strengths of Software Tools

Software tools can quickly identify objective problems such as images without alt text, form elements without <label> tags, tables without headers, and so on. Some tools, such as WAVE (www.wave.webaim.org - external link), can even identify a few of the more subjective problems, such as suspicious alt text, suspicious link text, and text that might be more appropriate as headings.

Several software tools can spider through web sites and produce reports for the entire site, including statistical analyses of the most frequent errors, a list of pages on which errors occur, and other useful information. Software tools perform these actions relatively quickly and tirelessly. Gathering this type of information without such software would be tedious, to say the least.

The Weaknesses of Software Tools

Software tools lack human judgment. Even though they can identify images missing alt text and some kinds of suspicious alt text, they are extremely limited in terms of subjective analysis. How can a tool know whether the alt text is accurate? What if the image is a picture of a baboon in a tree but the alt text says "Salmon heading upstream to spawn?" What if the image is a complex pie chart with lots of details, but the alt text says only "pie chart?" Only human evaluators can know if the alt text is accurate. Similar problems exist with such criteria as logical linearized reading order, understandable content, intuitive navigation, etc. Software tools are incapable of determining whether content is logical, understandable, or intuitive.

The Strengths of Human Evaluators

Humans make up for what software tools lack. Only humans can make the subjective judgment calls that make content not just "technically accessible" but truly usable and understandable. Humans can understand what other humans need much better than computer software can. Humans know the meanings behind words and symbols in ways that software simply can't. Despite all of the technological advances humans have made in computer software, humans still have their place. The place for humans is, in fact, at the very center of the design, development, and evaluation process. Web accessibility is for people, and for this reason, it is in many ways best accomplished by people.

The Weaknesses of Human Evaluators

It may be true that software tools lack human judgment, but so do some humans, unfortunately. Being able to identify accessibility barriers takes skill. It takes skill and experience to look at a web page and identify the problems for people with visual disabilities, hearing disabilities, motor disabilities, and cognitive disabilities. In this regard, untrained humans are at least as bad as unfeeling software programs. It takes time and effort to become an accessibility expert—or even to become familiar with the basics of web accessibility. Limited knowledge produces limited insights. Be careful in choosing a person as the human accessibility evaluator. Be diligent in ensuring that this person has the requisite skills, and that these skills are up-to-date.

People with Disabilities as Evaluators

The whole purpose of accessible web design is to make content accessible to people with disabilities. It makes sense, then, to get feedback from the people who will benefit the most. They can provide valuable insights into the ways that they access the web, the ways they use assistive technologies, the things they like about web pages, and the things they hate the most about web pages. If at all possible, developers should take advantage of the expertise of people with disabilities.

At the same time, feedback from people with disabilities can only be part of the picture. This may sound strange, but consider the type of feedback they would be giving. If developers ask a screen reader who is blind if they were able to access all of the features on the web site, there is no possible way that individual could answer this question with any sort of certainty. The most they could ever say would be that they accessed everything they were able to access. How would they know if parts of the web page were completely inaccessible to them? They can't ever know, because if any parts of the page are inaccessible, they couldn't get to them. It's like asking a blind person if she was able to meet everyone at a large banquet. Even if she met 132 people, how would she know that she missed someone without being told by someone else that she had missed that person?

With this in mind, it is usually best to evaluate web content along with the person with a disability. This allows developers to observe how they interact with the content, what parts of the content they missed, and what needs to be done to improve the user's experience. While observing people with disabilities accessing web content, it may be tempting to give them extra instructions ("No, the link's not there, it's at the end of the paragraph."), which won't give the observer a true sense of how easy or difficult it is for the user to navigate the content. It's hard to sit back and watch someone fumble through content that took weeks or months to create, but the more the user fumbles, the more the developer can learn from past mistakes.

The other problem with feedback from people with disabilities is that it can be tempting to think that the person giving the feedback speaks on behalf of all people with disabilities, or at least on behalf of all people with similar types of disabilities. Nobody can speak authoritatively about a whole class of people. It's like asking a white male to explain what white males are like. There is too much variation within the category of people to be able to say "that's what they're like."

For example, not all screen reader users are equally proficient with their screen reader software. Some are power users that can zip through pages at lightning speed. Others are so slow that it's almost painful to watch them. Some users with motor disabilities use a trackball mouse; others use the keyboard with a mouth stick; others use eye-tracking software—even among people with the same type of motor disability. No two people are alike, and no two people use the web in exactly the same way. Developers who design their web site to meet the needs of a user who is blind, or a user with dyslexia, may find that the page is no longer friendly to other users, with or without disabilities.

Nevertheless, do seek out feedback from people with disabilities. Do pay attention to what they say and do take their suggestions seriously. Just don't fall into the trap of thinking that feedback from one person with a disability is the same as controlled studies of representative populations of people with various types of disabilities.