WebAIM - Web Accessibility In Mind

E-mail List Archives

Re: Accessibility user testing

for

From: Chaals McCathie Nevile
Date: Jul 18, 2016 3:44AM

On Fri, 15 Jul 2016 21:16:49 +0200, Lucy Greco < <EMAIL REMOVED> > wrote:

> there is a lot to doing this but mostly you run it like any other user
> test i would not use hangouts i would use skipe if you can as it is
> easyer
> for the user to share screens with you and tends to be more accessable i
> run user tests like this all the time i am the screen reader user and my
> clients find this method works well lucy

The main idea is that I agree with Lucy - run the task exactly as you
would with any other user, and see what happens. Some more below…

> On Fri, Jul 15, 2016 at 12:09 PM, Zack McCartney
>> First off, hello all! This is my first post, excited to start learning
>> more about web accessibility.

(Welcome to the adventure of making the Web really work :) ).

>> Anyway, I work at a web development agency and I've been tasked with
>> running a usability test on a web application we've built with a
>> participant using a screenreader. Our development team just made a
>> bunch of updates to the site to move it closer to ADA (Americans with
>> Disabilities Act) compliance, so we're trying to find out if our first
>> pass actually improved the site's accessibility and what work still
>> needs to be done.

Hopefully it got closer. There is almost certainly room for improvement,
because this is reality :)

One thing to consider is the difference between "we complied with some
legal requirement, whose *goal* is to ensure that people with disabilities
can participate like anyone else", and another is "we're trying to make
sure that the site works for people with disabilities".

At a high level they are the same, but down in the details are important
but subtle differences. I think your *goal* should be the second one, but
understanding what happens when you aim strictly for the first will help
you learn a lot about accessibility, which often has both a "compliance"
or "legal" aspect, and a technical aspect of "actually solving the
problem".

>> The problem is: I've never run a usability test with a participant
>> using a screenreader. I have basic experience running usability
>> tests, so I have an ok handle on how to moderate a test session,
>> but I want to learn the basics of testing the user-friendliness of
>> web accessibility features.
>>
>> Specifically:
>>
>> - Do y'all have any advice on how to test the usability of a site's
>> accessibility features?

Check it with users who have disabilities - especially the ones you think
you targeted with some change you made.

Be aware that difference users have different skill levels, and different
tools - there is a recent thread on testing and what setups to use,
because different ones also behave differently. It's worth reading.

So when you've tested with one screenreader user, you know about one
screenreader user. If you're thinking of this as a professional
development exercise you need to consider questions like

"Does this user have a screenreader to support low vision, or because they
cannot see anything, or because they are dyslexic and the support of a
spoken version helps them, or for some other reason?"

"Does this person make a lot of effort, or rely on knowing their tools
very well, or does she just hope it works out and settle for whatever
result she got first?"

"Did I guide this user in a way I wouldn't normally do in a test? What
does that mean for the results?"

More along the lines of developing your expertise with regards to the web
in general, consider the improvements you tried to make and whether they
are applicable in general, or could you have changed something else to
make a general improvement that happened to help both a screenreader user
and everyone else.

And more generally for accessibility, learning a bit about the different
ways people with disabilities work to resolve problems will help you
design better tests as well as better solutions for end products. But you
need to remember whatever you did in this…

>> - What adaptations, if any, should I think to make to my typical
>> usability test setup?

As few as possible. Ideally nothing, but you may be relying on tools that
your tester cannot actually use properly. If that happens you need to
carefully identify those problems to figure out their impact on your data.

>> - The participant and I will be connecting over the phone, I'm
>> hoping over video call, with him sharing his screen. I have no idea if
>> this'll work or if asking him to navigate through a video conferencing
>> app (Google Hangouts) could complicate the test unnecessarily.

It depends on the user - in different setups, the environment has
different influence. Overall you probably need to learn this by
experiment, because I don't know of anyone who has published work that
explains this stuff.

If anyone has done so, or has material they *could* publish, it will help
a lot of people, so please speak up!

>> - Should I provide the participant instructions or can I (or rather,
>> typical of interacting with the web via screenreader) leave them in
>> the dark, let them figure out the site on their own?
>> - For a typical usability test, I'd want to the participant to
>> know as little as possible about the site under test, as I want to
>> learn how people figure out how to use a site on first encounter. But,
>> I don't know if omitting usage instructions — part of our dev team's
>> accessibility work — would prevent the user from even interacting with
>> the site. I want them to at the very least to be able to access the
>> site, even if it's still tricky to use on screenreader.

Start by keeping the information to describe the task at hand. This will
give the clearest information on what the user can do.

If you run into a complete block it might be more useful to help the user
over it to get more data - but you need to note what you did and how that
influenced the test. Which is nothing special to accessibility, of course.

>> Thanks!
>> Zack McCartney
>>
>> PS Sorry if my question shows my ignorance of web accessibility i.e.
>> anything sounds goofy or dumb. I'm totally new to the topic, trying to
>> get up to speed. :)

"There are no foolish questions, it is foolish not to ask questions"

cheers

Chaals

--
Charles McCathie Nevile - web standards - CTO Office, Yandex
<EMAIL REMOVED> - - - Find more at http://yandex.com