WebAIM - Web Accessibility In Mind

E-mail List Archives

Re: Accessibility user testing

for

From: Caitlin Geier
Date: Jul 26, 2016 2:01PM

I often do exactly what you've described. Basically, I set it up like a
standard usability test, except the tasks are specific to what a user would
need to learn in order to use the application successfully. For example, my
script for a task will briefly explain the terminology used for a
particularly feature, and what that feature was meant to be for. Then the
task script will go on to ask the user to accomplish a specific thing with
that feature ("now, create a new thing with this feature"). I ask them
questions about what they expect as they're performing tasks, and also take
time to answer their questions about what the expected functionality is
rather than letting them struggle to figure it out themselves. I do it this
way in part so that I can prime users with knowledge about the application
so I can come back to them later with more targeted questions about
specific functionality. Later sessions end up being a lot shorter when I
don't have to spend the time introducing them to the application first.

So yes, if the application is complex enough and specialized enough, it's
totally cool to use the same users more than once. Having a variety of
different people on tap to ask questions of - and who already know enough
about the application to be able to answer questions effectively - is
incredibly useful. The disadvantage is if you test with the same people
over and over, you cease to get the new user's perspective. Make sure you
continue to bring new users in for testing on a regular basis, even if it
does take longer to get them up to speed on the product. If you end up
building out an onboarding process, you'll definitely need to keep bringing
on new users to help test it out.

On Tue, Jul 26, 2016 at 3:19 PM, Zack McCartney < <EMAIL REMOVED> >
wrote:

> Hi all!
>
> *TLDR: What are you all's thoughts on using a guided tutorial / usability
> hybrid to test a highly dynamic, customizable web application to answer the
> question: **how do people first learn how to use this site? *
>
> First off, thank you to everyone that replied. I really appreciate all the
> help and advice. Definitely wasn't expecting so many and such in-depth
> responses. And sorry for not replying sooner; work's been a little mad
> lately :)
>
> Anyway, I'd like to pick this topic back up because I conducted the test
> about which I originally asked and have the fortune of being permitted to
> run another round of accessibility user tests next week.
>
> At the end of the first test, the participant offered to assist my company
> and I in further improving our site's accessibility features. He suggested
> maybe setting up a sort of guided tutorial, where I'd sit down with him and
> we'd walk through acquainting him with the site, showing him how to set it
> up, how to complete key tasks, etc.
>
> I wouldn't normally re-test a site with the same participant from a
> previous round of testing. However, I'm tempted to do so because it became
> unclear in the first test that the site under test — a dashboard of office
> organization and content tools for lawyers, basically, a heavily
> customizable, content-rich, super interactive site — doesn't seem to have a
> clear path to onboarding. It's not clear where people should start using
> it, let alone learning how to use it. My conclusion could be way off here,
> but I guess I'm thinking that without clearer context, as i'd provide in
> this sort of guided tutorial, the original test's tasks (although they did
> lead to some helpful insights) didn't make too much sense, given the site's
> highly custom nature.
>
> I'm envisioning this approach as a sort of exploratory usability test,
> prompting the participant to do things, but asking lots of questions along
> the way, having a conversation basically to uncover what they're thinking,
> what info they need to setup the site, what they hope and expect to be able
> to do with the site. I'm hoping that by talking the participant through
> what they need to know to setup the site, this tutorial format would help
> to answer: *how do people first learn how to use this site? what
> information do they need to even get started using this thing?*
>
> What are you all's thoughts on this sort of approach?
>
> Thanks!
> Zack
> > > > >



--
Caitlin Geier
User Experience Designer
<EMAIL REMOVED>