E-mail List Archives

Search E-mail List Archives for

Number of posts in this thread: 10 (In chronological order)

From: Amelia Dickerson
Date: Wed, Sep 02 2015 6:03PM
Subject: System Usability Scale with Blind Users
No previous message | Next message →

My question has 2 parts:
1. Has anyone used or seen anyone else use the System Usability Scale
http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
with blind/VI users when testing for accessibility/usability? Just
from a logical perspective, it seems like it would be just as valid
and reliable with a blind/VI user as with any other user-- it asks
about things like if you would want to use this system often and if
you are confident with it and if you felt able to figure it out
without training. At the same time, you aren't really supposed to
assume a tool is valid and reliable in a population where it hasn't
been tested. As a general rule, blind/VI users are probably going to
wind up with lower scores on usability than other users.

2. Is there another tool you use or have seen other people use to give
a general number on usability or accessibility when evaluating? We
are using WCAG guidelines and providing a lot of specifics. We also
try to offer a summary at the beginning. For the most part though, we
are communicating with people who know very little about
accessibility-- probably a bit more about general usability- so we are
trying to communicate in ways that make sense to them. Providing a
number might give a reader just one more way to try and relate to what
our reports say. A list of 20 problems can mean different things, but
having a score of 50 versus 80 is something that might help
communicate significance and impact and what it all winds up looking
like.
Thanks,
Amelia

From: Ron
Date: Wed, Sep 02 2015 7:28PM
Subject: Re: System Usability Scale with Blind Users
← Previous message | Next message →

I would be concerned about any scale that only looks at only one subset of
those with disability. This has been a pressing problem for a number of
years in the community.

Yes we need to deal with the issues of sensory disability, but I find much
more problematic the issues of alternative input, in particular Voice
Recognition tech.

We have been seeing for a number of years products that are accessible with
JAWS, the dominant product in the Screen Reader market. The problem is
these products fail more often than not in pan-disability evaluation. In
addition, they as a rule fail miserably in an accessibility and usability
evaluation.

Just some food for though.

On Wednesday, September 2, 2015, Amelia Dickerson < = EMAIL ADDRESS REMOVED = >
wrote:

> My question has 2 parts:
> 1. Has anyone used or seen anyone else use the System Usability Scale
>
> http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
> with blind/VI users when testing for accessibility/usability? Just
> from a logical perspective, it seems like it would be just as valid
> and reliable with a blind/VI user as with any other user-- it asks
> about things like if you would want to use this system often and if
> you are confident with it and if you felt able to figure it out
> without training. At the same time, you aren't really supposed to
> assume a tool is valid and reliable in a population where it hasn't
> been tested. As a general rule, blind/VI users are probably going to
> wind up with lower scores on usability than other users.
>
> 2. Is there another tool you use or have seen other people use to give
> a general number on usability or accessibility when evaluating? We
> are using WCAG guidelines and providing a lot of specifics. We also
> try to offer a summary at the beginning. For the most part though, we
> are communicating with people who know very little about
> accessibility-- probably a bit more about general usability- so we are
> trying to communicate in ways that make sense to them. Providing a
> number might give a reader just one more way to try and relate to what
> our reports say. A list of 20 problems can mean different things, but
> having a score of 50 versus 80 is something that might help
> communicate significance and impact and what it all winds up looking
> like.
> Thanks,
> Amelia
> > > > >

From: Karen Sorensen
Date: Fri, Sep 04 2015 11:33AM
Subject: Re: System Usability Scale with Blind/VI users
← Previous message | Next message →

Hi Amelia,
I haven't used this scale, but I'll check it out. At Portland Community
College, where I work, we do accessibility testing of information and
communication technologies used or being considered for use at the college.
But recently we have been discussing the difference between accessibility
and usability testing. I think what we do often is actually usability
testing. Because although something can be considered accessible by a
vendor, if an experienced screen reader user cannot actually make it work,
then we have a problem. It may be that documentation could fix the problem
instead of a technical change but it's still valid feedback. Accessibility
and usability testing should always include people with a wide range of
disabilities as Ron advises.
Thanks!
Karen
Karen M. Sorensen
Accessibility Advocate for Online Courses
www.pcc.edu/access
Portland Community College
971-722-4720

From: Whitney Quesenbery
Date: Tue, Sep 08 2015 9:49PM
Subject: Re: System Usability Scale with Blind Users
← Previous message | Next message →

The System Usability Scale is a simple 10-question scale for measuring
perceptions of usability. It's value is that it has been widely used over
many years, and validated by several different usability researchers,
including Jeff Sauro (Measuring Usability).

It poses statements about the use of a system (website, application, app,
etc) and asks the user to rate their agreement or disagreement with the
statement on a 5-point scale and uses that to produce an overall score.

It is technology neutral. It is not specific to any user group or any
disability/AT.

It can be used to compare SUS results by any demographic or usage
characteristic, but is more commonly aggregated for an entire product. It
would be useful to begin to gather SUS scores from a wider audience - as
anyone using it is free to do.

A few articles on the SUS from Jeff Sauro (and James Lewis)
http://uxpamagazine.org/sustified/
https://www.measuringu.com/blog/10-things-SUS.php
http://www.measuringu.com/sus.php
https://www.measuringu.com/topics/SUS (list of articles on the topic)It is
technology neutral

On Wed, Sep 2, 2015 at 9:28 PM Ron < = EMAIL ADDRESS REMOVED = > wrote:

> I would be concerned about any scale that only looks at only one subset of
> those with disability. This has been a pressing problem for a number of
> years in the community.
>
> Yes we need to deal with the issues of sensory disability, but I find much
> more problematic the issues of alternative input, in particular Voice
> Recognition tech.
>
> We have been seeing for a number of years products that are accessible with
> JAWS, the dominant product in the Screen Reader market. The problem is
> these products fail more often than not in pan-disability evaluation. In
> addition, they as a rule fail miserably in an accessibility and usability
> evaluation.
>
> Just some food for though.
>
> On Wednesday, September 2, 2015, Amelia Dickerson <
> = EMAIL ADDRESS REMOVED = >
> wrote:
>
> > My question has 2 parts:
> > 1. Has anyone used or seen anyone else use the System Usability Scale
> >
> >
> http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
> > with blind/VI users when testing for accessibility/usability? Just
> > from a logical perspective, it seems like it would be just as valid
> > and reliable with a blind/VI user as with any other user-- it asks
> > about things like if you would want to use this system often and if
> > you are confident with it and if you felt able to figure it out
> > without training. At the same time, you aren't really supposed to
> > assume a tool is valid and reliable in a population where it hasn't
> > been tested. As a general rule, blind/VI users are probably going to
> > wind up with lower scores on usability than other users.
> >
> > 2. Is there another tool you use or have seen other people use to give
> > a general number on usability or accessibility when evaluating? We
> > are using WCAG guidelines and providing a lot of specifics. We also
> > try to offer a summary at the beginning. For the most part though, we
> > are communicating with people who know very little about
> > accessibility-- probably a bit more about general usability- so we are
> > trying to communicate in ways that make sense to them. Providing a
> > number might give a reader just one more way to try and relate to what
> > our reports say. A list of 20 problems can mean different things, but
> > having a score of 50 versus 80 is something that might help
> > communicate significance and impact and what it all winds up looking
> > like.
> > Thanks,
> > Amelia
> > > > > > > > > >
> > > > >

From: Tim Harshbarger
Date: Tue, Sep 08 2015 9:52PM
Subject: Re: System Usability Scale with Blind/VI users
← Previous message | Next message →

SUS is definitely something that could be used as part of usability testing with people of any type of disability.

However, the one thing I would urge that you keep in mind is that it measures the user's perception of the system's usability--which is not necessarily the same as the reality of how usable the system is. That isn't to say that there is no value in using SUS. It is just important to keep in mind what SUS really measures.

I tend to think of accessibility and usability as a weird type of pyramid. At the bottom you have accessibility. If you don't have accessibility, you can't even really begin to address usability. From a task oriented perspective, until the tasks are accessible, you can't really even start to address issues such as accuracy or efficiency.

The weird aspect to the pyramid is that it isn't built of bricks. There isn't a layer of accessibility bricks at the bottom with usability bricks placed on the layer above. There isn't a clear cut division between usability and accessibility.

It is kind of like dawn. There is a point when it is clearly night. And there is a point when it is clearly day. But there is a transition between when it might be argued whether it is still day or night. I hope that makes some kind of sense.

Hopefully something I wrote is useful food for thought.

From: Tim Harshbarger
Date: Tue, Sep 08 2015 9:54PM
Subject: Re: System Usability Scale with Blind Users
← Previous message | Next message →

Whitney,

I was hoping you might comment on SUS. Thanks for doing so!

From: Ron
Date: Wed, Sep 09 2015 6:24AM
Subject: Re: System Usability Scale with Blind Users
← Previous message | Next message →

Appreciate the info my evaluation team will definitely take a look.

Ron Stewart

On Tuesday, September 8, 2015, Tim Harshbarger <
= EMAIL ADDRESS REMOVED = > wrote:

> Whitney,
>
> I was hoping you might comment on SUS. Thanks for doing so!
>
>

From: Tim Harshbarger
Date: Wed, Sep 09 2015 10:57AM
Subject: Re: System Usability Scale with Blind Users
← Previous message | Next message →

For those of you, who are thinking about using SUS to enhance your accessibility work, I would definitely urge you to share your results. Particularly with regards to how it works for you, what benefits you find in using it, what kind of pitfalls people should be aware of when they use it, etc.

From: Whitney Quesenbery
Date: Sat, Sep 12 2015 2:16PM
Subject: Re: System Usability Scale with Blind Users
← Previous message | Next message →

The other value to sharing results is that it will give a better sense of
baselines. Tom Tullis, Jeff Sauro and Jim Lewis did a rather large-scale
collection that led to Jeff's publication of baseline scores.

It would be awesome to be able to add more data, collected with users with
disabilities.

However, before someone jumps on this, I am NOT suggesting (in fact would
oppose) a "disability baseline". There is no reason to do so, and many
reasons not to.

However, I could see a project that could measure things like satisfaction
among a wide variety of users before and after a major accessibility
remediation.

If anyone is interested in pursuing this, I'd be happy to connect you to
Jeff or Tom, so that the data can be aggregated with their previous work,
giving us a richer, integrated, and even a little longitudinal view.



On Wed, Sep 9, 2015 at 12:57 PM Tim Harshbarger <
= EMAIL ADDRESS REMOVED = > wrote:

> For those of you, who are thinking about using SUS to enhance your
> accessibility work, I would definitely urge you to share your results.
> Particularly with regards to how it works for you, what benefits you find
> in using it, what kind of pitfalls people should be aware of when they use
> it, etc.
>
>

From: Whitney Quesenbery
Date: Sat, Sep 12 2015 2:26PM
Subject: Re: System Usability Scale with Blind/VI users
← Previous message | No next message

Agree with Tim's clarification and what he said about usability and
accessibility. I fall back on the ISO definition of accessibility as
usability for people with the widest range of capabilities. And usability
as efficiency, effectiveness, and satisfaction in reaching a goal.

So it's not enough to have a site (page, group of pages, function...) that
is technically correct. It must support users in doing something - finding
info or taking an action.

The importance of satisfaction in this equation is that it can balance
absolute measures, especially for efficiency. If all you measured was time
on task, you would never know that even though a task might take a while to
complete (even longer for one user than another), it can still be
satisfying if the user had previously not been able to do it at all.

And it might be quick but if it doesn't get the job done, it not only fails
effectiveness, but is also likely to produce low satisfaction.

Just remember that there is a recency bias, and that satisfaction is mixed
up with expectations for ease. Example that I see all the time: a user is
learning how to use an interface for a task they think ought to be easy.
After a few tries they figure it out, and rapidly increase in efficiency
and effectiveness. Satisfaction is likely to be rates as high, if they
learn how to use the interface in a length of time they feel is appropriate
- and early struggles tend to be forgotten. So timing of satisfaction
questions make a difference.

Others reasons to measure perceived satisfaction (and related, trust) are:
- Increases in satisfaction can often accompany improvements in the
usability of the most common tasks
- Users are more likely to persist in working through problems on sites
which they perceive as being good




On Tue, Sep 8, 2015 at 11:52 PM Tim Harshbarger <
= EMAIL ADDRESS REMOVED = > wrote:

> SUS is definitely something that could be used as part of usability
> testing with people of any type of disability.
>
> However, the one thing I would urge that you keep in mind is that it
> measures the user's perception of the system's usability--which is not
> necessarily the same as the reality of how usable the system is. That
> isn't to say that there is no value in using SUS. It is just important to
> keep in mind what SUS really measures.
>
> I tend to think of accessibility and usability as a weird type of
> pyramid. At the bottom you have accessibility. If you don't have
> accessibility, you can't even really begin to address usability. From a
> task oriented perspective, until the tasks are accessible, you can't really
> even start to address issues such as accuracy or efficiency.
>
> The weird aspect to the pyramid is that it isn't built of bricks. There
> isn't a layer of accessibility bricks at the bottom with usability bricks
> placed on the layer above. There isn't a clear cut division between
> usability and accessibility.
>
> It is kind of like dawn. There is a point when it is clearly night. And
> there is a point when it is clearly day. But there is a transition between
> when it might be argued whether it is still day or night. I hope that
> makes some kind of sense.
>
> Hopefully something I wrote is useful food for thought.
>
>