8-Step Implementation Model
Step 1: Gather Baseline Information

The Need to Know

How bad is your web site?

How much of your web site is accessible?

How do you know?

How would you find out?

Even among people who are informed about web accessibility issues, many of them can only take a guess at how much of their web site is accessible. They don't have accurate data with respect to the accessibility status of their site. Some of them have never thought about formally analyzing the site. Many of them do not know how to do so. Without knowing this information, an organization will find it hard to effectively address the problem.

"My web site must be OK, because no one has ever complained about it."

Some people with disabilities actively pursue issues of equality by voicing their opinions and making their needs known. The majority, however, do not. Most of them are so accustomed to encountering accessibility barriers that they have become passive about the whole issue, figuring that it is more bother than it is worth to try to get others to accommodate their needs. Even if no one has filed an official complaint, this is not confirmation that the web site is accessible. The only way to know with certainty if a web site is accessible or not is to test it. In this workshop, we'll discuss how to perform those tests.

What to Test

The question of what to test depends largely on the type of web site to be tested. Small, uncomplicated web sites created by a single person are the easiest to test. The most difficult sites to test are large, complex ones, such as those at universities, where there are tens, if not hundreds, of developers. The testing strategies are similar, but there are enough differences that it is beneficial to distinguish between the strategies for testing different kinds of web sites.

Home page

The home page is the top-most level of a web site, and is usually the "root" of the web address. For example, the home page of the main WebAIM site is webaim.org. All of the other areas of the web site are usually available from the home page, either directly or indirectly. If the home page is inaccessible, users will likely not get to any other part of the web site.

Top level pages

The top level pages of a site can usually be found by clicking on the largest, or the most noticeable links on the home page, such as the main navigation items. Top level pages represent the way in which the information of a web site has been categorized, meaning that all other pages are organized under these top-level "headings".

Most often, they are found at the top of the page or on the left side, but may also be found on the right side (as is the case in this example). If the top-level pages have been thoughtfully planned out, they represent the major conceptual areas of the web site. After the home page, these are usually the most accessed pages on a web site.

With this in mind, the pages that are linked from these top-level links are a high priority in terms of accessibility. If the top-level pages are inaccessible, users will likely not be able to get to the subsequent document pages, which probably contain the content that they're really trying to access.

Document pages

After the home page and top-level links, many web sites have at least one extra layer of document pages. These pages represent the actual content of the site, more than either the home page or top-level links. Usually the home page and top-level pages are merely gateways to the more detailed information on the document pages.

Users generally have to pass through the home page and the top-level links in order to arrive at the content that they want to access. Once they arrive at the document page, if that page is inaccessible, then they will not be able to access its content.

Simple web sites

In the context of this discussion, a simple web site is one that has fewer than about 50 pages, and which does not employ much scripting or multimedia content. Many personal home pages fit this profile, as do sites by some small businesses, non-profit organizations, government agencies, and other organizations.

If a web site is small enough, it is possible to evaluate all of its pages in a relatively short amount of time. Our recommendation would be to do just that. Start with the home page, then proceed to each of the top-level links, then through all of the document pages. If the site approaches 50 pages, then it is probably not necessary to evaluate each and every page at first. Usually a pattern emerges, and the remedies become apparent after evaluating only a few of the content pages.

Large and/or complex web sites

Large and/or complex sites, such as e-commerce sites, university sites, large government agencies, and so on require special attention. The types of tasks that users perform on the site may extend beyond just reading text and viewing graphics. In many cases, complex sites are more accurately viewed as software applications, much like software that one would install on the hard drive. These sites (sometimes referred to as "web applications") can be quite sophisticated, presenting the possibility for even more accessibility errors and barriers.

Scripted sites/ Web applications

Other than the home page, top-level links, and document pages, the first step with web applications is to determine where the important interactions take place. If users must fill in a form to access the web application, this is a critical component that must be checked for accessibility. An e-commerce site must ensure that the entire process of browsing, selecting and purchasing items is accessible. It is not enough to claim that the home page is accessible. The complete process must be accessible, so this is one of the first things that must be analyzed. Similarly, a tax preparation web site cannot claim to be accessible unless all of the procedures and instructions can be completed by people with disabilities. The key with web applications is to determine what the most important tasks are that the user will perform, and analyze the tasks in that order.

Decentralized sites

University web sites are the perfect example of decentralized sites. Universities have a main home page with basic information about the university, but there are also home pages for each of the colleges or departments, for the employee information, for student registration, and a host of other purposes. In essence, universities have multiple ancillary or secondary web sites that, although related, are often completely different from each other.

In situations like this, the ideal would be to evaluate the main university web site plus all of the ancillary web sites. Coordinators could prioritize a list of ancillary web sites that are most critical for accessibility (e.g. student registration), and evaluate these sites during the first phase of evaluation. Later phases could address other sites lower down on the priority list.

When evaluating each of these ancillary sites, the same basic procedure applies: evaluate the home page, then the top-level pages, then a sample of the document pages. This can be quite a time-consuming process if there are many high-priority ancillary sites, but it is necessary.

Important

Test the home page first, then the top-level pages, then a representative sample of the document pages. This may need to be repeated on ancillary or secondary web sites that are related to, but separate from the main web site.

Accessibility Validators

Choosing an evaluation tool

We will not offer any specific recommendations for evaluation tools here, because each tool is different, with its own strengths and weaknesses in comparison to the others. In many ways it is simply a matter of personal preference. Most evaluation tools are available in a trial version, so you can check them out on your own time and decide which is best for you and your particular situation.

The usefulness of evaluation tools

Without a doubt, one of the quickest ways to get a feel for the accessibility of a web site is to run it through an evaluation tool. Some of these tools are free, some expensive; some only give a report, while others help the user to fix the errors; some analyze one page at a time, others crawl through entire web sites, saving time when evaluating large sites. Without this kind of software, it would take a person much longer to determine whether or not a page meets accessibility guidelines. These tools flag errors quickly and efficiently.

The limitations of evaluation tools

Software tools for evaluating web accessibility are useful when evaluating web sites in the same way that a spell checkers can be useful in evaluating written documents. Both of these also share some of the same weaknesses. For instance, an image of a button that says "products" might have alt text that says "services." None of the tools will be able to catch this error. They all will notice that alt text is present, but all are incapable of determining whether or not the alt text is appropriate.

Recording the results

Most tools allow the user to choose which standard against which to evaluate the web pages. At a minimum, most tools allow the user to choose between the Web Content Accessibility Guidelines and The US Federal Government standard, Section 508. Some tools also let the user ignore certain types of errors, which can result in a more focused report. If there is no official web accessibility standard at an organization, it may be wise to generate reports based on more than one standard. To be considered accessible to any reasonable degree, all web sites should pass at least WCAG 2.0 Level A, and Section 508.

Don't be too surprised if few or none of the pages of your organization "pass" these software evaluation tools. Unfortunately, that's normal. In this case, though, normal is not desirable, but that's why you're performing this evaluation. You'll have ample opportunity to remedy the situation as you proceed through the steps outlined in this article.

But before you conclude that you really know how accessible your site is, it is best to perform a second evaluation, this time by a human being.

Human Judgment

The need for knowledgeable human judgment

Until a knowledgeable human being checks a web page for accessibility, there will always be doubts. It is entirely possible for a site to pass all of the standards and guidelines and still present significant accessibility challenges to some individuals with disabilities. There are at least a couple of reasons for this. First, the guidelines and standards themselves are imperfect. They do not contain every piece of advice that is necessary for all kinds of disabilities under all circumstances. Secondly, Sometimes the guidelines are misunderstood and misapplied. It takes a knowledgeable person to be able to identify this type of error.

That raises a question: where and who are these knowledgeable people? Hopefully you'll be able to qualify as a knowledgeable person after working with web accessibility, but sometimes the person in the role of coordinator is not the best person to judge web pages. If you are not technologically knowledgeable enough to design accessible web content, then you are probably not the best person to evaluate the accessibility of web content. Try to find a web developer at your organization who is familiar enough with the techniques of web accessibility to be able to identify errors that the tools do not catch. If you are the only one at your organization with any kind of knowledge in this area, then you'll either have to perform this task yourself (whether you are qualified or not), or hire a consultant who can perform this task.

What to judge

Automated tools are able to evaluate web pages more quickly than human beings are. It may not be possible or even beneficial to evaluate every single page of a web site this way, at least not while trying to establish a baseline. Pick some of the most important pages to evaluate, and then randomly pick a few of the less important pages. It would be ideal to evaluate all of the same pages that were previously evaluated with the automated software tool, but even a smaller sample of pages will be informative.

A tool for judging pages

When we at WebAIM evaluate pages, we make extensive use of the WAVE evaluation tool because this tool is specifically designed for this task. The WAVE does not give a "pass" or "fail" rating. Rather, it presents information about the page in the context where the elements occur. The WAVE inserts icons and text which advise the use to take a closer look at what is present on the page.

Despite the strengths of the WAVE, evaluators must never lose sight of the fact that WAVE and other evaluators are only tools. It can never substitute for intelligent interpretation.

Using a checklist

When trying to determine a baseline level of accessibility, it isn't enough to just peruse through the site using WAVE or some other tool. The people performing the evaluation must document their findings. One of the best ways to do this is by using a checklist. WebAIM has put together a WCAG 2.0 Checklist and a Section 508 checklist. Both of these checklists can be useful when gathering information.

If more than one person is performing the evaluations, it is important to ensure high inter-rater reliability. The people using the checklist should be familiar with web accessibility concepts. They should independently rate a test group of pages, then compare notes and opinions. The human evaluation of web pages will be effective only if it is clear that these people will rate pages similarly.

It helps to plan ahead when using a checklist. If the raters enter the data directly into a spreadsheet or a database, it is much easier to generate reports later.

In many cases it makes sense to have a "not applicable" category. For example, if you are evaluating a page according to Section 508 standards, you will have to determine whether "equivalent alternatives for any multimedia presentation [are] synchronized with the presentation." Perhaps none of your pages have any multimedia at all. If that is the case, you would mark the "not applicable" category for this item. Just make sure that you are consistent. Don't mark this item as "not-applicable" on one page and "pass" on another page. This will make your data more difficult to interpret.

Once a checklist has been filled out, it must be interpreted, as explained in the next section.

Interpreting the Results

It will probably be helpful to generate a few different types of reports. For example, it can be helpful to compare the pass/fail ratio of pages evaluated with the automated software to the pass/fail ratio of pages evaluated by human evaluators. You could also separate out the errors by type, WAI priority or other criteria.

When comparing the results of the two different methods of evaluation, you may find that the errors detected by the automated tools are not serious, and that they can be corrected quite easily. On the other hand, you may find that the automated tools could not detect the fact that all of the important content was presented in a video clip that did not have captions or a transcript. Human evaluation of the pages will reveal these potential discrepancies.

Here is a simplified example of a summary report that could be generated:

A sample summary report

Accessibility Evaluation of Gidget's Groovy Gizmos, Inc.

Total number of pages tested by automated tools: 408
Total number of pages tested by human evaluators: 50

Pages that passed the automated evaluation tools: 34 (8%)
Pages that passed the human evaluators: 14 (3%)

Most common types of errors according to the automated tools:

  1. Missing alt text: 21,390 instances on 380 pages
  2. Missing form labels: 312 instances on 35 pages
  3. Missing frame titles: 12 instances on 6 pages

Most common types of errors according to the human evaluators:

  1. Missing alt text: on 48 pages (96%)
  2. Missing form labels: on 15 pages (30%)
  3. Missing table headers: on 9 pages (18%)

Average number of errors per page:

  • according to the automated tools: 21
  • according to the human evaluators: 27

Areas of the site with the most errors:

home page, top-level pages


Overall rating: Poor

Recommended action: Start immediately to fix these errors!

End of report.

This kind of summary will make it easier in the future to compare how far your organization has progressed as a whole.

You may also generate reports for specific pages, such as the home page, top-level pages, and a few of the document pages. A copy of the report could be given to the developer in charge of those pages, with recommendations on how to fix the errors.