WebAIM - Web Accessibility In Mind

E-mail List Archives

RE: Accessibility Testing questions

for

From: Mark Magennis
Date: Aug 2, 2002 2:37AM

Susan,

Good to hear you are doing some real testing. The main bit of advice I would
give you is to be present during the tests. I know you said you would likely
not be present, but you will gain so much more by seeing issues arise first
hand that you really should try, if at all possible, to be there. Other than
that, I would definitely recommend task-based tests. This will ensure that
the issues that arise are the ones that would arise during real use. It will
also help prevent you unwittingly directing the tests towards the areas of
the site that you expect the issues to arise in and missing other areas
completely. Testing always brings up issues you didn't expect and task-based
testing is the best way to do this.

As for how many users - how long is a piece of string? It is a basic fact of
usability testing that to test any software product thoroughly enough to
ensure a 95% chance of finding all the usability problems would cost more
than the entire software development budget for the next five years. At
least! It is simply not feasible to do "complete" usability testing. Nobody
ever does it. People used to believe in Jakob Nielsen's study which showed
that you needed 5 users before getting into seriously diminishing returns
for revealing problems. That's no longer accepted. More recent studies have
shown that the more users you have, the more issues you keep finding. Also,
the more observers you have, the more issues you uncover. It has even been
shown that different test teams come up with almost entirely different
results. What happens in reality is that you do as much as you have time
for. No more and no less. Then you content yourself that you have done as
good a test as you can.

Try to embrace a wide variety of users, particularly including blind,
partially sighted and fully-sighted keyboard-only users. Older people make
great test participants.

What sorts of assistive technologies they use probably doesn't matter. You
won't be able to cover them all and you probably shouldn't try to bias it to
any specific ones.

I would also advise having two observers - yourself and someone else. The
observers will undoubtedly disagree with each other, not only about how to
interpret what happened, but even about what actually happened! Two minds
arguing about issues and interpretation are much better than one.

Payment to users varies. I used to work in the commercial field where we
always paid 65 Euro for an hour and a half, plus travelling expenses. I now
work in a non-commercial arena where it has not been normal to make any
payment, other than travel expenses.

Hope this helps,

Mark

>