Rocket Surgery and Accessibility User Testing

When people ask us about accessibility user testing, we usually say, “Don’t do it.” Instead, usability testing with users with disabilities is almost always more effective.

Rocket Surgery Made Easy

I spoke at the Plain Talk conference last week where I heard presentations on usability testing from Steve Krug and Nicole Burton. Steve’s book, Rocket Surgery Made Easy, proposes a basic approach to usability testing wherein frequent (probably monthly), simple usability evaluation sessions are conducted with three average users. A facilitator asks questions and presents tasks to help the user identify the few most significant usability issues, which are then (hopefully) fixed before the next test session.

The power of the Rocket Surgery approach to usability testing is that it focuses on the broad user experience. Accessibility user testing typically does just the opposite. It is used to identify instances of inaccessibility – poor alt text here and a missing label there. Fixing all significant instances of inaccessibility and non-compliance still might result in a poor experience for users with disabilities – something often a result of the entire site content/interaction or a combination of many small issues, rather than glaring WCAG failures.

When users with disabilities are involved in effective usability testing, such as the basic, low-cost approach Steve recommends, they will help address the overall accessibility and usability of a site, and will still be able to identify specific accessibility issues.

Gotchas

While general user testing can be conducted very early in the design process, accessibility testing will likely work better on a site where some accessibility has already been implemented – certainly any WAVE errors addressed, a WCAG checklist reviewed, etc. A usability testing session with a screen reader user, for example, might be very short and ineffective when you realize that your fancy, yet totally inaccessible interface results in no content being read. A lesson learned, for sure, but nothing as valuable as having this user describe their screen reader interactions on a more polished site.

Be careful – while people with disabilities are good at finding accessibility issues, if something is inaccessible, they might not be able to identify it because it’s inaccessible. Unlike usability issues, accessibility issues are often caused by underlying markup or compatibility issues, making them more difficult to identify. You’ll probably want someone versed in accessibility and assistive technology to monitor or facilitate the usability session.

You’ll usually want the users to use their own computer systems, particularly if they are using assistive technology. Recording and monitoring the session will still be possible – even if the user is remotely located.

Should I Stop Doing Accessibility User Testing?

Steve notes in his book, “There‚Äôs no question: a good usability professional will be able to do a better job of testing than you will.” This also applies to accessibility. WebAIM, and other experts, are very good at this type of thing. Accessibility and usability testing are both vital, regardless of who does it or how it’s done. If you want to conduct the testing yourself, a low-budget, easy to facilitate, highly effective usability test that includes individuals with disabilities will result in more useful feedback and a more accessible web site than conducting distinct accessibility user testing.

Comments

  1. Pooja

    Completely agree that PwDs wouldnt be able to find inaccessibility issues – as firstly that piece of code is not available to their AT so finding the issue in that code is not realistic. Thus I prefer to a functional testing using developers/consultant with expertise in technology, AT, Ax guidelines, and an understanding on how PwDs interact with the web.

    But this doesn’t mean elimination of testing with PwDs – they are the users – so a scenario/use case based testing works best to get effective results. But yes, there may be cases where assistance is required by PwDs to conduct scenario based testing.

    To conclude – a product unless tested by PwD cannot be called accessible!

  2. Shawn (uiAccess)

    Here are some resources that provide more info on some of the points mentioned:

    * Guidance on involving users even earlier in the design process: “Involving Users in Web Projects for Better, Easier Accessibility” http://www.w3.org/WAI/users/involving.html

    * “Involving Users in Evaluating Web Accessibility” http://www.w3.org/WAI/eval/users

    * http://uiAccess.com/JustAsk includes chapters on usability testing with participants with disabilities

  3. G F Mueden

    After the accessibility and usability work is done, give the viewer a way to feedback that tells you the problem, what he is using, and if he still uses his eyes, his needs for magnification and contrast. Not being able to provide direct feedback is terribly annoying. Doing it through a big website (Microsoft, Adobe, Canon) or by snailmail (Google) doesn’t work for the disabled. Most of us give up.

  4. Kevin

    One on one monitoring of a user using the site is always best. Group monitoring sessions or focus groups tend to be skewed in favour of the person who is most vocal or opinionated.