WebAIM - Web Accessibility In Mind

E-mail List Archives

Re: System Usability Scale with Blind Users

for

From: Whitney Quesenbery
Date: Sep 12, 2015 2:16PM

The other value to sharing results is that it will give a better sense of
baselines. Tom Tullis, Jeff Sauro and Jim Lewis did a rather large-scale
collection that led to Jeff's publication of baseline scores.

It would be awesome to be able to add more data, collected with users with
disabilities.

However, before someone jumps on this, I am NOT suggesting (in fact would
oppose) a "disability baseline". There is no reason to do so, and many
reasons not to.

However, I could see a project that could measure things like satisfaction
among a wide variety of users before and after a major accessibility
remediation.

If anyone is interested in pursuing this, I'd be happy to connect you to
Jeff or Tom, so that the data can be aggregated with their previous work,
giving us a richer, integrated, and even a little longitudinal view.



On Wed, Sep 9, 2015 at 12:57 PM Tim Harshbarger <
<EMAIL REMOVED> > wrote:

> For those of you, who are thinking about using SUS to enhance your
> accessibility work, I would definitely urge you to share your results.
> Particularly with regards to how it works for you, what benefits you find
> in using it, what kind of pitfalls people should be aware of when they use
> it, etc.
>
> -----Original Message-----
> From: WebAIM-Forum [mailto: <EMAIL REMOVED> ] On
> Behalf Of Ron
> Sent: Wednesday, September 09, 2015 7:24 AM
> To: WebAIM Discussion List
> Subject: Re: [WebAIM] System Usability Scale with Blind Users
>
> Appreciate the info my evaluation team will definitely take a look.
>
> Ron Stewart
>
> On Tuesday, September 8, 2015, Tim Harshbarger <
> <EMAIL REMOVED> > wrote:
>
> > Whitney,
> >
> > I was hoping you might comment on SUS. Thanks for doing so!
> >
> > -----Original Message-----
> > From: WebAIM-Forum [mailto: <EMAIL REMOVED>
> > <javascript:;>] On Behalf Of Whitney Quesenbery
> > Sent: Tuesday, September 08, 2015 10:49 PM
> > To: WebAIM Discussion List
> > Subject: Re: [WebAIM] System Usability Scale with Blind Users
> >
> > The System Usability Scale is a simple 10-question scale for measuring
> > perceptions of usability. It's value is that it has been widely used over
> > many years, and validated by several different usability researchers,
> > including Jeff Sauro (Measuring Usability).
> >
> > It poses statements about the use of a system (website, application, app,
> > etc) and asks the user to rate their agreement or disagreement with the
> > statement on a 5-point scale and uses that to produce an overall score.
> >
> > It is technology neutral. It is not specific to any user group or any
> > disability/AT.
> >
> > It can be used to compare SUS results by any demographic or usage
> > characteristic, but is more commonly aggregated for an entire product.
> It
> > would be useful to begin to gather SUS scores from a wider audience - as
> > anyone using it is free to do.
> >
> > A few articles on the SUS from Jeff Sauro (and James Lewis)
> > http://uxpamagazine.org/sustified/
> > https://www.measuringu.com/blog/10-things-SUS.php
> > http://www.measuringu.com/sus.php
> > https://www.measuringu.com/topics/SUS (list of articles on the topic)It
> is
> > technology neutral
> >
> > On Wed, Sep 2, 2015 at 9:28 PM Ron < <EMAIL REMOVED>
> <javascript:;>>
> > wrote:
> >
> > > I would be concerned about any scale that only looks at only one subset
> > of
> > > those with disability. This has been a pressing problem for a number of
> > > years in the community.
> > >
> > > Yes we need to deal with the issues of sensory disability, but I find
> > much
> > > more problematic the issues of alternative input, in particular Voice
> > > Recognition tech.
> > >
> > > We have been seeing for a number of years products that are accessible
> > with
> > > JAWS, the dominant product in the Screen Reader market. The problem is
> > > these products fail more often than not in pan-disability evaluation.
> In
> > > addition, they as a rule fail miserably in an accessibility and
> usability
> > > evaluation.
> > >
> > > Just some food for though.
> > >
> > > On Wednesday, September 2, 2015, Amelia Dickerson <
> > > <EMAIL REMOVED> <javascript:;>>
> > > wrote:
> > >
> > > > My question has 2 parts:
> > > > 1. Has anyone used or seen anyone else use the System Usability Scale
> > > >
> > > >
> > >
> >
> http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
> > > > with blind/VI users when testing for accessibility/usability? Just
> > > > from a logical perspective, it seems like it would be just as valid
> > > > and reliable with a blind/VI user as with any other user-- it asks
> > > > about things like if you would want to use this system often and if
> > > > you are confident with it and if you felt able to figure it out
> > > > without training. At the same time, you aren't really supposed to
> > > > assume a tool is valid and reliable in a population where it hasn't
> > > > been tested. As a general rule, blind/VI users are probably going to
> > > > wind up with lower scores on usability than other users.
> > > >
> > > > 2. Is there another tool you use or have seen other people use to
> give
> > > > a general number on usability or accessibility when evaluating? We
> > > > are using WCAG guidelines and providing a lot of specifics. We also
> > > > try to offer a summary at the beginning. For the most part though,
> we
> > > > are communicating with people who know very little about
> > > > accessibility-- probably a bit more about general usability- so we
> are
> > > > trying to communicate in ways that make sense to them. Providing a
> > > > number might give a reader just one more way to try and relate to
> what
> > > > our reports say. A list of 20 problems can mean different things,
> but
> > > > having a score of 50 versus 80 is something that might help
> > > > communicate significance and impact and what it all winds up looking
> > > > like.
> > > > Thanks,
> > > > Amelia
> > > > > > > > > > > > > > > > > > <javascript:;>
> > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > >