WebAIM - Web Accessibility In Mind

E-mail List Archives

Re: Accessibility user testing

for

From: Caitlin Geier
Date: Jul 18, 2016 6:51AM

A few pieces of advice, from my own experience running usability tests with
screen reader users:

- Hangouts or Skype. They're the most accessible tools allowing screen
sharing that I've found. I also use Snagit to record sessions, since it can
be used with a variety of communication tools.
- If the user you work with has trouble sharing his/her screen, have a
backup plan! A few times when this has happened, I've asked the user to
tell me where they are on the page as they're going through it so I can
follow along on my own screen. It's actually helpful in general to ask a
screen reader user to tell you what their screen reader is saying so that
you know where they are on the page. Since most screen reader users don't
use a mouse, there's no cursor or pointer to follow. Most screen reader
users also don't travel linearly through a page.
- It's helpful to ask which OS, browser, and screen reader they're using
up front. Screen readers, like browsers, all have their quirks.
- Plan extra time, or plan to skip a task or two if you want to keep to
a time limit, especially if the site you're testing is very content-heavy.

Also, if you ever do any semi-automated usability testing (i.e. where you
set up the test and volunteers go through it at their leisure - think
UserTesting.com), AccessWorks
<http://www.access-works.knowbility.org/index.php>; is a good service
specifically for testing with users with disabilities. You can set up a
test via Loop11 and they will recruit users with whatever disabilities you
specify for you to test with.

-Caitlin

On Mon, Jul 18, 2016 at 5:44 AM, Chaals McCathie Nevile <
<EMAIL REMOVED> > wrote:

> On Fri, 15 Jul 2016 21:16:49 +0200, Lucy Greco < <EMAIL REMOVED> >
> wrote:
>
> there is a lot to doing this but mostly you run it like any other user
>> test i would not use hangouts i would use skipe if you can as it is easyer
>> for the user to share screens with you and tends to be more accessable i
>> run user tests like this all the time i am the screen reader user and my
>> clients find this method works well lucy
>>
>
> The main idea is that I agree with Lucy - run the task exactly as you
> would with any other user, and see what happens. Some more below…
>
> On Fri, Jul 15, 2016 at 12:09 PM, Zack McCartney
>>
>>> First off, hello all! This is my first post, excited to start learning
>>> more about web accessibility.
>>>
>>
> (Welcome to the adventure of making the Web really work :) ).
>
> Anyway, I work at a web development agency and I've been tasked with
>>> running a usability test on a web application we've built with a
>>> participant using a screenreader. Our development team just made a bunch
>>> of updates to the site to move it closer to ADA (Americans with
>>> Disabilities Act) compliance, so we're trying to find out if our first
>>> pass actually improved the site's accessibility and what work still
>>> needs to be done.
>>>
>>
> Hopefully it got closer. There is almost certainly room for improvement,
> because this is reality :)
>
> One thing to consider is the difference between "we complied with some
> legal requirement, whose *goal* is to ensure that people with disabilities
> can participate like anyone else", and another is "we're trying to make
> sure that the site works for people with disabilities".
>
> At a high level they are the same, but down in the details are important
> but subtle differences. I think your *goal* should be the second one, but
> understanding what happens when you aim strictly for the first will help
> you learn a lot about accessibility, which often has both a "compliance" or
> "legal" aspect, and a technical aspect of "actually solving the problem".
>
> The problem is: I've never run a usability test with a participant using a
>>> screenreader. I have basic experience running usability
>>> tests, so I have an ok handle on how to moderate a test session,
>>> but I want to learn the basics of testing the user-friendliness of
>>> web accessibility features.
>>>
>>> Specifically:
>>>
>>> - Do y'all have any advice on how to test the usability of a site's
>>> accessibility features?
>>>
>>
> Check it with users who have disabilities - especially the ones you think
> you targeted with some change you made.
>
> Be aware that difference users have different skill levels, and different
> tools - there is a recent thread on testing and what setups to use, because
> different ones also behave differently. It's worth reading.
>
> So when you've tested with one screenreader user, you know about one
> screenreader user. If you're thinking of this as a professional development
> exercise you need to consider questions like
>
> "Does this user have a screenreader to support low vision, or because they
> cannot see anything, or because they are dyslexic and the support of a
> spoken version helps them, or for some other reason?"
>
> "Does this person make a lot of effort, or rely on knowing their tools
> very well, or does she just hope it works out and settle for whatever
> result she got first?"
>
> "Did I guide this user in a way I wouldn't normally do in a test? What
> does that mean for the results?"
>
> More along the lines of developing your expertise with regards to the web
> in general, consider the improvements you tried to make and whether they
> are applicable in general, or could you have changed something else to make
> a general improvement that happened to help both a screenreader user and
> everyone else.
>
> And more generally for accessibility, learning a bit about the different
> ways people with disabilities work to resolve problems will help you design
> better tests as well as better solutions for end products. But you need to
> remember whatever you did in this…
>
> - What adaptations, if any, should I think to make to my typical
>>> usability test setup?
>>>
>>
> As few as possible. Ideally nothing, but you may be relying on tools that
> your tester cannot actually use properly. If that happens you need to
> carefully identify those problems to figure out their impact on your data.
>
> - The participant and I will be connecting over the phone, I'm
>>> hoping over video call, with him sharing his screen. I have no idea if
>>> this'll work or if asking him to navigate through a video conferencing
>>> app (Google Hangouts) could complicate the test unnecessarily.
>>>
>>
> It depends on the user - in different setups, the environment has
> different influence. Overall you probably need to learn this by experiment,
> because I don't know of anyone who has published work that explains this
> stuff.
>
> If anyone has done so, or has material they *could* publish, it will help
> a lot of people, so please speak up!
>
> - Should I provide the participant instructions or can I (or rather,
>>> typical of interacting with the web via screenreader) leave them in
>>> the dark, let them figure out the site on their own?
>>> - For a typical usability test, I'd want to the participant to
>>> know as little as possible about the site under test, as I want to learn
>>> how people figure out how to use a site on first encounter. But,
>>> I don't know if omitting usage instructions — part of our dev team's
>>> accessibility work — would prevent the user from even interacting with
>>> the site. I want them to at the very least to be able to access the
>>> site, even if it's still tricky to use on screenreader.
>>>
>>
> Start by keeping the information to describe the task at hand. This will
> give the clearest information on what the user can do.
>
> If you run into a complete block it might be more useful to help the user
> over it to get more data - but you need to note what you did and how that
> influenced the test. Which is nothing special to accessibility, of course.
>
> Thanks!
>>> Zack McCartney
>>>
>>> PS Sorry if my question shows my ignorance of web accessibility i.e.
>>> anything sounds goofy or dumb. I'm totally new to the topic, trying to
>>> get up to speed. :)
>>>
>>
> "There are no foolish questions, it is foolish not to ask questions"
>
> cheers
>
> Chaals
>
> --
> Charles McCathie Nevile - web standards - CTO Office, Yandex
> <EMAIL REMOVED> - - - Find more at http://yandex.com
>
> > > > >



--
Caitlin Geier
User Experience Designer
<EMAIL REMOVED>