Rocket Surgery and Accessibility User Testing

When people ask us about accessibility user testing, we usually say, “Don’t do it.” Instead, usability testing with users with disabilities is almost always more effective.

Rocket Surgery Made Easy

I spoke at the Plain Talk conference last week where I heard presentations on usability testing from Steve Krug and Nicole Burton. Steve’s book, Rocket Surgery Made Easy, proposes a basic approach to usability testing wherein frequent (probably monthly), simple usability evaluation sessions are conducted with three average users. A facilitator asks questions and presents tasks to help the user identify the few most significant usability issues, which are then (hopefully) fixed before the next test session.

The power of the Rocket Surgery approach to usability testing is that it focuses on the broad user experience. Accessibility user testing typically does just the opposite. It is used to identify instances of inaccessibility – poor alt text here and a missing label there. Fixing all significant instances of inaccessibility and non-compliance still might result in a poor experience for users with disabilities – something often a result of the entire site content/interaction or a combination of many small issues, rather than glaring WCAG failures.

When users with disabilities are involved in effective usability testing, such as the basic, low-cost approach Steve recommends, they will help address the overall accessibility and usability of a site, and will still be able to identify specific accessibility issues.

Gotchas

While general user testing can be conducted very early in the design process, accessibility testing will likely work better on a site where some accessibility has already been implemented – certainly any WAVE errors addressed, a WCAG checklist reviewed, etc. A usability testing session with a screen reader user, for example, might be very short and ineffective when you realize that your fancy, yet totally inaccessible interface results in no content being read. A lesson learned, for sure, but nothing as valuable as having this user describe their screen reader interactions on a more polished site.

Be careful – while people with disabilities are good at finding accessibility issues, if something is inaccessible, they might not be able to identify it because it’s inaccessible. Unlike usability issues, accessibility issues are often caused by underlying markup or compatibility issues, making them more difficult to identify. You’ll probably want someone versed in accessibility and assistive technology to monitor or facilitate the usability session.

You’ll usually want the users to use their own computer systems, particularly if they are using assistive technology. Recording and monitoring the session will still be possible – even if the user is remotely located.

Should I Stop Doing Accessibility User Testing?

Steve notes in his book, “There’s no question: a good usability professional will be able to do a better job of testing than you will.” This also applies to accessibility. WebAIM, and other experts, are very good at this type of thing. Accessibility and usability testing are both vital, regardless of who does it or how it’s done. If you want to conduct the testing yourself, a low-budget, easy to facilitate, highly effective usability test that includes individuals with disabilities will result in more useful feedback and a more accessible web site than conducting distinct accessibility user testing.