WebAIM - Web Accessibility In Mind

E-mail List Archives

Re: Accessibility user testing

for

From: David Sloan
Date: Jul 18, 2016 8:44AM

To add to the excellent advice already shared on usability testing in this thread, something that might seem stating the obvious but is still worth emphasising: don't waste the time of people with disabilities in finding barriers that you could find through manual or automated inspection.

In other words, you have a responsibility to your organisation and to your participants to make best use of the time you have to gather data on task flow and other, rich, user experience aspects, from the perspective of people with disabilities. This might include understanding the cumulative effect of apparently minor issues on successful task completion.

If there are known issues, design the tasks so that you're not unnecessarily exposing participants to these issues, by helping them bypass them—in the words of David Hamill, "give participants a punty over the wall" [1]

Dave

[1] https://www.youtube.com/watch?v=OMGA4CQUPKU (video of a talk on general tips on running usability studies; unfortunately this video doesn't yet have captions and the auto captions are not helpful at all)


> On 18 Jul 2016, at 13:51, Caitlin Geier < <EMAIL REMOVED> > wrote:
>
> A few pieces of advice, from my own experience running usability tests with
> screen reader users:
>
> - Hangouts or Skype. They're the most accessible tools allowing screen
> sharing that I've found. I also use Snagit to record sessions, since it can
> be used with a variety of communication tools.
> - If the user you work with has trouble sharing his/her screen, have a
> backup plan! A few times when this has happened, I've asked the user to
> tell me where they are on the page as they're going through it so I can
> follow along on my own screen. It's actually helpful in general to ask a
> screen reader user to tell you what their screen reader is saying so that
> you know where they are on the page. Since most screen reader users don't
> use a mouse, there's no cursor or pointer to follow. Most screen reader
> users also don't travel linearly through a page.
> - It's helpful to ask which OS, browser, and screen reader they're using
> up front. Screen readers, like browsers, all have their quirks.
> - Plan extra time, or plan to skip a task or two if you want to keep to
> a time limit, especially if the site you're testing is very content-heavy.
>
> Also, if you ever do any semi-automated usability testing (i.e. where you
> set up the test and volunteers go through it at their leisure - think
> UserTesting.com), AccessWorks
> <http://www.access-works.knowbility.org/index.php>; is a good service
> specifically for testing with users with disabilities. You can set up a
> test via Loop11 and they will recruit users with whatever disabilities you
> specify for you to test with.
>
> -Caitlin
>
> On Mon, Jul 18, 2016 at 5:44 AM, Chaals McCathie Nevile <
> <EMAIL REMOVED> > wrote:
>
>> On Fri, 15 Jul 2016 21:16:49 +0200, Lucy Greco < <EMAIL REMOVED> >
>> wrote:
>>
>> there is a lot to doing this but mostly you run it like any other user
>>> test i would not use hangouts i would use skipe if you can as it is easyer
>>> for the user to share screens with you and tends to be more accessable i
>>> run user tests like this all the time i am the screen reader user and my
>>> clients find this method works well lucy
>>>
>>
>> The main idea is that I agree with Lucy - run the task exactly as you
>> would with any other user, and see what happens. Some more below…
>>
>> On Fri, Jul 15, 2016 at 12:09 PM, Zack McCartney
>>>
>>>> First off, hello all! This is my first post, excited to start learning
>>>> more about web accessibility.
>>>>
>>>
>> (Welcome to the adventure of making the Web really work :) ).
>>
>> Anyway, I work at a web development agency and I've been tasked with
>>>> running a usability test on a web application we've built with a
>>>> participant using a screenreader. Our development team just made a bunch
>>>> of updates to the site to move it closer to ADA (Americans with
>>>> Disabilities Act) compliance, so we're trying to find out if our first
>>>> pass actually improved the site's accessibility and what work still
>>>> needs to be done.
>>>>
>>>
>> Hopefully it got closer. There is almost certainly room for improvement,
>> because this is reality :)
>>
>> One thing to consider is the difference between "we complied with some
>> legal requirement, whose *goal* is to ensure that people with disabilities
>> can participate like anyone else", and another is "we're trying to make
>> sure that the site works for people with disabilities".
>>
>> At a high level they are the same, but down in the details are important
>> but subtle differences. I think your *goal* should be the second one, but
>> understanding what happens when you aim strictly for the first will help
>> you learn a lot about accessibility, which often has both a "compliance" or
>> "legal" aspect, and a technical aspect of "actually solving the problem".
>>
>> The problem is: I've never run a usability test with a participant using a
>>>> screenreader. I have basic experience running usability
>>>> tests, so I have an ok handle on how to moderate a test session,
>>>> but I want to learn the basics of testing the user-friendliness of
>>>> web accessibility features.
>>>>
>>>> Specifically:
>>>>
>>>> - Do y'all have any advice on how to test the usability of a site's
>>>> accessibility features?
>>>>
>>>
>> Check it with users who have disabilities - especially the ones you think
>> you targeted with some change you made.
>>
>> Be aware that difference users have different skill levels, and different
>> tools - there is a recent thread on testing and what setups to use, because
>> different ones also behave differently. It's worth reading.
>>
>> So when you've tested with one screenreader user, you know about one
>> screenreader user. If you're thinking of this as a professional development
>> exercise you need to consider questions like
>>
>> "Does this user have a screenreader to support low vision, or because they
>> cannot see anything, or because they are dyslexic and the support of a
>> spoken version helps them, or for some other reason?"
>>
>> "Does this person make a lot of effort, or rely on knowing their tools
>> very well, or does she just hope it works out and settle for whatever
>> result she got first?"
>>
>> "Did I guide this user in a way I wouldn't normally do in a test? What
>> does that mean for the results?"
>>
>> More along the lines of developing your expertise with regards to the web
>> in general, consider the improvements you tried to make and whether they
>> are applicable in general, or could you have changed something else to make
>> a general improvement that happened to help both a screenreader user and
>> everyone else.
>>
>> And more generally for accessibility, learning a bit about the different
>> ways people with disabilities work to resolve problems will help you design
>> better tests as well as better solutions for end products. But you need to
>> remember whatever you did in this…
>>
>> - What adaptations, if any, should I think to make to my typical
>>>> usability test setup?
>>>>
>>>
>> As few as possible. Ideally nothing, but you may be relying on tools that
>> your tester cannot actually use properly. If that happens you need to
>> carefully identify those problems to figure out their impact on your data.
>>
>> - The participant and I will be connecting over the phone, I'm
>>>> hoping over video call, with him sharing his screen. I have no idea if
>>>> this'll work or if asking him to navigate through a video conferencing
>>>> app (Google Hangouts) could complicate the test unnecessarily.
>>>>
>>>
>> It depends on the user - in different setups, the environment has
>> different influence. Overall you probably need to learn this by experiment,
>> because I don't know of anyone who has published work that explains this
>> stuff.
>>
>> If anyone has done so, or has material they *could* publish, it will help
>> a lot of people, so please speak up!
>>
>> - Should I provide the participant instructions or can I (or rather,
>>>> typical of interacting with the web via screenreader) leave them in
>>>> the dark, let them figure out the site on their own?
>>>> - For a typical usability test, I'd want to the participant to
>>>> know as little as possible about the site under test, as I want to learn
>>>> how people figure out how to use a site on first encounter. But,
>>>> I don't know if omitting usage instructions — part of our dev team's
>>>> accessibility work — would prevent the user from even interacting with
>>>> the site. I want them to at the very least to be able to access the
>>>> site, even if it's still tricky to use on screenreader.
>>>>
>>>
>> Start by keeping the information to describe the task at hand. This will
>> give the clearest information on what the user can do.
>>
>> If you run into a complete block it might be more useful to help the user
>> over it to get more data - but you need to note what you did and how that
>> influenced the test. Which is nothing special to accessibility, of course.
>>
>> Thanks!
>>>> Zack McCartney
>>>>
>>>> PS Sorry if my question shows my ignorance of web accessibility i.e.
>>>> anything sounds goofy or dumb. I'm totally new to the topic, trying to
>>>> get up to speed. :)
>>>>
>>>
>> "There are no foolish questions, it is foolish not to ask questions"
>>
>> cheers
>>
>> Chaals
>>
>> --
>> Charles McCathie Nevile - web standards - CTO Office, Yandex
>> <EMAIL REMOVED> - - - Find more at http://yandex.com
>>
>> >> >> >> >>
>
>
>
> --
> Caitlin Geier
> User Experience Designer
> <EMAIL REMOVED>
> > > > David Sloan

UX Research Lead
The Paciello Group
<EMAIL REMOVED>

CONFIDENTIALITY NOTICE: The information contained in this message may be privileged, confidential and protected from disclosure. If you are not the intended recipient, any use, disclosure, dissemination, distribution or copying of any portion of this message or any attachment is strictly prohibited. If you think you have received this message in error, please notify the sender at the above e-mail address, and delete this e-mail along with any attachments. Thank you.