WebAIM - Web Accessibility In Mind

E-mail List Archives

RE: formalized 508 testing process.

for

From: Paul Bohman
Date: Nov 7, 2002 2:18PM

Here's an approach that we've used before:

Step 1. Evaluate the page (or site) using an automated tool, such as
Bobby (but any autmated tool will do here)
Step 2. Evaluate the page (or site) by human beings using a checklist.

The two approaches are necessary to get reliable and accurate results.
Automated tools are excellent for catching missing alt text, missing
frame titles, and other "yes or no" type items. Human testers are better
at evaluating the quality of the accessibility features, and humans are
the only ones that can test for some of the items, such as the document
linearization (i.e. whether the document displays in a logical reading
order when tables and CSS are removed).

The key to quality human testing is to
1. use or create a clear checklist (see
http://www.webaim.org/standards/508/checklist for one example of a
section 508 checklist)
2. talk about how certain forseeable circumstances will be judged (e.g.
if there are frame titles but the titles are inadequate, will you pass
or fail this item? [I would fail it] or if a site provides a text only
version even though the main version could have been made accessible,
does this page still pass section 508 by virtue of [k] in Section 508?
[My answer would be "no"] )
3. If you have multiple raters, make sure that they agree on how to rate
sites, by having them separately rate a few test sites and then
comparing their ratings. Also, it would be ideal to have the raters
re-evaluate some of the sites which their co-raters evaluated, to check
for inter-rater reliability.

Of course, it is important to have raters who know what they're doing.
Your raters should have a good background in HTML and accessibility.

So that is what I would recommend: automated tool plus human testers.
The truth is that either method alone will always be inadequate.


By the way, when doing human testing, we commonly use the WAVE validator
to check for the appropriateness of alt text, because the WAVE displays
the alt text right next to the image, making the evaluation easy and
quick to perform. We use the Opera browser to check for page
linearization, because you can turn the tables off in "user mode" and
then toggle between user mode and document mode with one click of the
mouse and easily evaluate whether or not the page linearizes well when
tables are turned off (Opera can also turn of CSS layout with one click
of the mouse).

Paul Bohman
Technology Coordinator
WebAIM (Web Accessibility in Mind)
www.webaim.org
Center for Persons with Disabilities
www.cpd.usu.edu
Utah State University
www.usu.edu