WebAIM - Web Accessibility In Mind

E-mail List Archives

Re: Accessibility user testing

for

From: Steve Green
Date: Jul 19, 2016 12:30PM

Some people have mentioned running remote sessions using screen sharing. I'm not at all keen on that. I like to see what keystrokes participants are using and what they are doing with the mouse. I also like to see their expression and body language - it tells you so much more than their words alone. Because of this, we used to do all our user testing in people's homes although we now do quite a lot in our in-house UX lab.

Whilst I agree that the focus should generally be on whether people are successful in completing the scenarios you give them, bear in mind that it only shows that the scenario *can* be completed successfully. It does not mean that all users with the identical disability *will* be able to complete it. So much depends on their preferred strategies for navigating websites and individual pages. Many times I have seen people complete tasks on websites that we considered to have poor accessibility, just because the accessible parts of the website matched their preferred strategies.

For this reason I recommend supplementing user testing with an expert review with assistive technologies that methodically goes through all the content. Neither is a complete solution but together they are about as good as you can get (assuming you have already done WCAG compliance testing to achieve a good level of technical accessibility).

Steve Green
Test Partners Ltd


-----Original Message-----
From: WebAIM-Forum [mailto: <EMAIL REMOVED> ] On Behalf Of Birkir R. Gunnarsson
Sent: 19 July 2016 15:30
To: WebAIM Discussion List < <EMAIL REMOVED> >
Subject: Re: [WebAIM] Accessibility user testing

Just one more thought. If you are not familiar with screen reading, you might want to try and find an expert screen reader user, even an accessibility person, in your area and do a pilot test with that person face to face.
I think it may help you set expectations and get a little bit of a feel for the task.
You can be successful without it, but I often find that, as a screen reader user, sitting down with usability testers and going through the tasks informally helps them feel more comfortable.
Cheers



On 7/19/16, Tim Harshbarger < <EMAIL REMOVED> > wrote:
> This is one reason why the usability test should focus on tasks and
> not just single components.
>
> Ultimately, everyone who uses an interface is trying to complete some
> kind of task or achieve some kind of goal. Some components of the
> interface might be accessible and others might be inaccessible. But
> what matters to the user is whether or not they can complete the task.
>
> So, in the example, the search box evidently was an accessibility
> disaster while other parts of the site were fine. However, if the
> search box prevents users from completing any other task, then the
> amazing accessibility of the rest of the interface does not matter--at
> least not until the search box is fixed.
>
> So, you probably want to talk about the results of a usability test in
> relation to how those accessibility barriers impacted user
> tasks--rather than talk about each accessibility problem as though it
> were a separate self-contained thing.
>
>
>
>
>
>
> -----Original Message-----
> From: WebAIM-Forum [mailto: <EMAIL REMOVED> ] On
> Behalf Of Moore,Michael (Accessibility) (HHSC)
> Sent: Tuesday, July 19, 2016 8:33 AM
> To: WebAIM Discussion List < <EMAIL REMOVED> >
> Subject: Re: [WebAIM] Accessibility user testing
>
> Your example also shows why a small problem can create a really big barrier.
> I could see a development manager commenting on the report. Well 99%
> of the site is accessible. It's just the search box that is the
> problem and Google indexes our site so fixing the search is a low
> priority" Meanwhile because of the problem with the search people
> using screen readers can't get to the "accessible" part of the site.
>
> Mike Moore
> Accessibility Coordinator
> Texas Health and Human Services Commission Civil Rights Office
> (512) 438-3431 (Office)
>
> -----Original Message-----
> From: WebAIM-Forum [mailto: <EMAIL REMOVED> ] On
> Behalf Of L Snider
> Sent: Monday, July 18, 2016 5:29 PM
> To: WebAIM Discussion List < <EMAIL REMOVED> >
> Subject: Re: [WebAIM] Accessibility user testing
>
> Hi Zack,
>
> For your question about advice on barriers...I never provided advice
> to testers, because in the real world the user would maybe try a
> couple of things that they had done before as workarounds, but if they
> failed they couldn't use the site, and would need to go away from it.
> For my user testing and research study, I only gave tasks to testers,
> and recording what happened. I didn't want my biases or knowledge to
> get in the way. For me, I did it this way because it was then done the
> same way for every tester and I had no 'bias' in the process. I felt
> this was important in terms of the end results, so that they were 'real world', albeit in a testing realm.
>
> I just wanted to add one thing that happened when I did a research
> study that involved user testing by people using different AT. I ran
> into a case where a screen reader user could not get on to the page
> due to a problematic search box that was at the top of the code and it
> meant the user could not keyboard anywhere on the page to complete the
> task. It was a mess, because that search box kept them out of the
> website as a whole (I used that site a lot to show people what not to
> do). So that person could not complete the task. You may encounter
> this, and it is a powerful weapon to show why inaccessible websites should not be in the world in 2016!
>
> Cheers
>
> Lisa
> Lisa Snider
> Accessibility Consultant
> Everything Accessibility
>
> On Mon, Jul 18, 2016 at 4:40 PM, Zack McCartney
> < <EMAIL REMOVED> >
> wrote:
>
>> Hi Dana,
>>
>> First off, thanks so much for your detailed reply and all the advice.
>> And for the resources, too. This sort of test is totally outside my
>> comfort zone, so I really appreciate the starting points :)
>>
>> I wanted to follow up on your point about providing assistance to the
>> participant: do you have any advice for how to assist and instruct
>> participants when the encounter barriers?
>>
>> I'm unsure about doing this well in usability testing in general,
>> even more so with participants using a screenreader.
>>
>> Thanks again for all your help and welcoming me to the forum!
>> Zack
>>
>>
>> On Mon, Jul 18, 2016 at 11:06 AM Dana Douglas
>> < <EMAIL REMOVED> >
>> wrote:
>>
>> > Zack,
>> >
>> > Welcome! Including people with disabilities in a usability test is
>> > a
>> great
>> > way to evaluate a site's accessibility. Here are a few resources
>> > you may find helpful (there are certainly others as well) :
>> >
>> > Slides from my 2015 UXPA presentation on the topic:
>> >
>> http://www.slideshare.net/UXPA/uxpa-2015-why-how-to-include-people-wi
>> t
>> h-disabilities-pw-ds-in-your-usability-testsdouglas-and-davis-6252015
>> -
>> 49918852
>> > A UXPA Magazine article by Mary Hunter Utt on the topic:
>> > http://uxpamagazine.org/guerilla_tactics/
>> > An article from Deque Systems on the topic:
>> > http://www.deque.com/blog/incorporate-users-disabilities-ux-testing
>> > /
>> >
>> > In general, the test can be virutally the same as any other
>> > usability
>> test
>> > (same tasks, methodology, etc.). You will want to make sure the web
>> > conference tool you're using is accessible for screen reader users
>> > (and other assistive technologies). Skype may be the best bet -
>> > participants
>> can
>> > share their screen through Skype. The test should be as realistic
>> > as possible. If users would not have any outside instructions or
>> > information in the real world, you should not provide that
>> > information during the usabilty test either. However, if that
>> > presents a significant barrier (first of all, you know you have an
>> > accessibility issue to fix!), but
>> then
>> > you can provide assistance to move on and gather additional
>> > feedback on other aspects of the site.
>> >
>> > Good luck!
>> >
>> > Dana Douglas
>> >
>> > -----Original Message-----
>> > From: Zack McCartney [mailto: <EMAIL REMOVED> ]
>> > Sent: Friday, July 15, 2016 3:10 PM
>> > To: <EMAIL REMOVED>
>> > Subject: [WebAIM] Accessibility user testing
>> >
>> > Hi there!
>> >
>> > First off, hello all! This is my first post, excited to start
>> > learning more about web accessibility.
>> >
>> > Anyway, I work at a web development agency and I've been tasked
>> > with running a usability test on a web application we've built with
>> > a participant using a screenreader. Our development team just made
>> > a bunch
>> of
>> > updates to the site to move it closer to ADA (Americans with
>> > Disabilities
>> > Act) compliance, so we're trying to find out if our first pass
>> > actually improved the site's accessibility and what work still
>> > needs to be done.
>> >
>> > The problem is: I've never run a usability test with a participant
>> > using
>> a
>> > screenreader. I have basic experience running usability tests, so I
>> > have
>> an
>> > ok handle on how to moderate a test session, but I want to learn
>> > the
>> basics
>> > of testing the user-friendliness of web accessibility features.
>> >
>> > Specifically:
>> >
>> > - Do y'all have any advice on how to test the usability of a site's
>> > accessibility features?
>> >
>> >
>> > - What adaptations, if any, should I think to make to my typical
>> > usability test setup?
>> > - The participant and I will be connecting over the phone,
>> > I'm
>> hoping
>> > over video call, with him sharing his screen. I have no idea
>> > if this'll
>> > work or if asking him to navigate through a video
>> > conferencing app (Google
>> > Hangouts) could complicate the test unnecessarily.
>> >
>> >
>> > - Should I provide the participant instructions or can I (or rather,
>> > typical of interacting with the web via screenreader) leave them
>> > in
>> the
>> > dark, let them figure out the site on their own?
>> > - For a typical usability test, I'd want to the participant
>> > to know
>> > as little as possible about the site under test, as I want to
>> > learn how
>> > people figure out how to use a site on first encounter. But,
>> > I
>> don't
>> > know
>> > if omitting usage instructions — part of our dev team's
>> > accessibility work
>> > — would prevent the user from even interacting with the site.
>> > I
>> want
>> > them
>> > to at the very least to be able to access the site, even if
>> > it's still
>> > tricky to use on screenreader.
>> >
>> >
>> > Thanks!
>> > Zack McCartney
>> >
>> > PS Sorry if my question shows my ignorance of web accessibility i.e.
>> > anything sounds goofy or dumb. I'm totally new to the topic, trying
>> > to
>> get
>> > up to speed. :)
>> >
>> >
>> >> >> archives at http://webaim.org/discussion/archives
>> >>
> > > archives at http://webaim.org/discussion/archives
> > > > archives at http://webaim.org/discussion/archives
> > > > archives at http://webaim.org/discussion/archives
> >


--
Work hard. Have fun. Make history.