E-mail List Archives

Search E-mail List Archives for

Number of posts in this thread: 14 (In chronological order)

From: Aaron Cannon
Date: Thu, Mar 13 2008 8:40AM
Subject: accessibility without testing?
No previous message | Next message →

Hi all.

It has always been my understanding that in order to consistently create accessible pages, particularly pages which are accessible to screen reader users, it is necessary to conduct testing with assistive technology. I.E. before you can say for sure something is accessible, someone has to go check. Is this view accurate? Is there a more automated way of ensuring accessibility that I'm unaware of?

I know that there have been some products marketed to be able to do so, but I was under the impression that they were just snake oil. I also know of a couple products, which are quite good at identifying potential problem areas, but that they can't possibly identify them all, nor are they immune to false-positives.

Any information would be appreciated.

Thanks.

Aaron Cannon

----------------------------------------------------------------------
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

From: Christian Heilmann
Date: Thu, Mar 13 2008 8:50AM
Subject: Re: accessibility without testing?
← Previous message | Next message →

> It has always been my understanding that in order to consistently create accessible pages, particularly pages which are accessible to screen reader users, it is necessary to conduct testing with assistive technology. I.E. before you can say for sure something is accessible, someone has to go check. Is this view accurate? Is there a more automated way of ensuring accessibility that I'm unaware of?
>
>
Would you release a Japanese version of your site without having a
native speaker have a read-through?

Accessibility is a human interaction matter, not a technical problem.
Automated testing tools can flag up *big* technical issues but cannot
tell you about semantic problems or wrong text.

From: Tom Babinszki
Date: Thu, Mar 13 2008 9:00AM
Subject: Re: accessibility without testing?
← Previous message | Next message →

Hi Aaron,

If I understand your question correctly, you would like to know if you
should use assistive technology to test for accessibility, and also if there
is a way to test a page automatically, without human intervention.

It depends what you are testing for, what do you want to be your page
accessible with. If you want to comply with an accessibility standard or
guideline, such as Section 508 or WCAG then testing with assistive
technology, such as JAWS is almost irrelevant, because you will have to
comply with a set of rules which you have to follow regardless of
technology. These standards are made so they are accessible for all people,
including all types of disabilities, using all kinds of assistive
technology. This sounds way too perfect, because the reality is a bit
different. Just because there is an accessibility regulation, it does not
mean that assistive technology works well with it. For example, consider
JAWS and AJAX. According to the WCAG you can create perfectly accessible
AJAX based products, but the JAWS screen reader might not read it properly.
In this case, the screen reader manufacturers should put more effort into
making their applications work with a set of internationally accepted and
use guidelines. However, in the meantime, blind people will have
difficulties using it. So, once you satisfied the accessibility standards
you have picked, depending on your page, it might be a good idea to ensure
that it works with a screen reader.

For testing with tools only: Look at the following page for more
information:
http://www.evengrounds.com/web/articles/article1.shtml
I'll soon have something on the site about testing with assistive
technology, as well.

But the short answer is the following:
Nowadays there are some very sophisticated tools which will help you test
for accessibility. But if you look at any of the standards or guidelines, it
will always require human interaction. For example, when you label a
picture. A testing tool will tell you if a picture has a label. It could
also tell you that it has a null tag (empty label), because it is a
decorative image, or it has a longer label, which contains a description.
But a testing tool will never be able to tell you if the description is
grammatically correct, if it makes sense, etc. And this is just one example.
I would use testing tools and add your personal judgment.

Let me know if I can help you with more details.

Tom


Tom Babinszki, PMP
Even Grounds
Accessibility consulting
E-mail: = EMAIL ADDRESS REMOVED =
Phone: +1 (703) 853-2990
http://www.evengrounds.com

From: Gareth Dart
Date: Thu, Mar 13 2008 9:10AM
Subject: Re: accessibility without testing?
← Previous message | Next message →

"It has always been my understanding that in order to consistently
create accessible pages, particularly pages which are accessible to
screen reader users, it is necessary to conduct testing with assistive
technology. I.E. before you can say for sure something is accessible,
someone has to go check. Is this view accurate? Is there a more
automated way of ensuring accessibility that I'm unaware of?"

Several people have already pointed out that automated accessibility
testing is only part of a proper accessibility testing framework. It
is, however, better than the alternative all too many designers still
take, which is to do almost nothing, either because of a lack of
awareness of web accessibility issues, or a lack of resources to
undertake testing. In such cases it's far preferable that a designer
runs his pages through an automated accessibility testing tool of some
kind and corrects at least the most glaring accessibility errors rather
than publish the page 'as is'. It would be great if they went on from
that point to do proper testing, but where this isn't possible then the
benefits brought around by automated tools shouldn't be understated.

Cheers,

Gareth

From: Travis Smith
Date: Thu, Mar 13 2008 9:20AM
Subject: Re: accessibility without testing?
← Previous message | Next message →

As an Individual who works for a software developer, and a user of JAWS.
I completely agree that nothing will replace good old fashion human
testing. Because no matter how good a software gets it will never to
have the leave of understanding and learning ability that the human
brain has. Therefore, it will not catch enough of the problems to be
able to say that a site is accessible.

From: Oscar DeLong
Date: Thu, Mar 13 2008 9:30AM
Subject: Re: accessibility without testing?
← Previous message | Next message →

I would argue that it is always best to test it. Even if you find a
program that can automate the process it can never replace field
testing. In part, testing the site first helps one to determine if
wording, I.E. bad or confusing phrases, and order are appropriate,
something an automated tool can never do. I would use tools to assist
in the process but still test it to make sure.

Oscar DeLong
Access Services/Reference Librarian
Ottenheimer Library
Pulaski Technical College
3000 West Scenic Drive
North Little Rock, AR 72118
(501) 812-2718
= EMAIL ADDRESS REMOVED =

From: Steve Green
Date: Thu, Mar 13 2008 9:40AM
Subject: Re: accessibility without testing?
← Previous message | Next message →

Firstly, testing for compliance with the WCAG only tells you if a website
'should' be accessible. It does not tell you that it 'is' accessible.
Automated tools can only make a decision about 25% of the WCAG checkpoints,
so a significant degree of manual testing would be required to verify WCAG
compliance.

To be sure a site 'is' accessible, you would need to conduct user testing
with the user groups you are concerned about, taking into account the wide
variations in users' level of ability and the various user agents and
assistive tchnologies they use. That is invariably prohibitively expensive.

In short, automated testing, manual testing and user testing provide
increasing levels of confidence that a site is accessible but you can't have
absolute certainty.

Steve

From: Aaron Cannon
Date: Thu, Mar 13 2008 12:40PM
Subject: Re: accessibility without testing?
← Previous message | Next message →

Thanks all for the response. We're trying to define some accessibility standards for our organization, and this topic came up in discussion.

Thanks again.

Aaron

----------------------------------------------------------------------
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

From: Karl Groves
Date: Thu, Mar 13 2008 3:50PM
Subject: Re: accessibility without testing?
← Previous message | Next message →

I would say that yes, the root of what we're after here is whether a product
can be used by disabled users, so the "true" measure of accessibility is
found by having the product tested by disabled users.

At the same time, there is a LOT that can be discovered through other means.
Automated testing can be used to go through a lot of code very quickly to
find items which frankly don't need humans to find.
Manual review of the code can be used to validate items uncovered during the
automated testing and find items the automated testing wasn't able to find.
User testing would be used as a means to uncover anything not found through
the other methods and to validate the results of the other two methods.

It is important to keep in mind that JAWS is not the only assistive
technology used by disabled users. Further, different versions of JAWS may
deal with things differently. Last, the experience level of the person
testing comes into play. "User testing" with JAWS by a sighted newbie is
next to worthless, IMO.


Karl Groves
AIM/YIM: karlcore
Skype: eight.pistons
www.WebAccessStrategies.com


>

From: John Foliot - Stanford Online Accessibility Program
Date: Thu, Mar 13 2008 5:30PM
Subject: Re: accessibility without testing?
← Previous message | Next message →

Tom Babinszki wrote:
> For example, consider JAWS and AJAX. According to the WCAG
> you can create perfectly accessible AJAX based products,

Really? Can you kindly point out where in WCAG it states this? Having
worked with this guideline since it's release in 1999 (back when AJAX was a
cleaning product), I would love to know how you reached this conclusion.

> but the JAWS
> screen reader might not read it properly. In this case, the screen
> reader manufacturers should put more effort into making their
> applications work with a set of internationally accepted and use
> guidelines.

Huh? At best, what we have today is ARIA, for which we have an incomplete
but growing series of test-case documents which illustrate how the various
ARIA roles and states should behave. But ARIA, and it's implementation is
pretty cutting-edge, and as far as I know about the only screen-reading type
application that currently plays nice with ARIA is FireVox plugin for
Firefox. You also neatly side-stepped the whole issue of *versioning*: many
people reliant on adaptive technology such as JAWS cannot always afford to
upgrade with every release; at $1,000 +/- per computer, it is often out of
reach for many who (sadly) struggle just to keep their heads above the
poverty line. So while it might be easy to blame the developers of screen
reading technology, the issue is way more complicated than stating that they
"..should put more effort in..."

> However, in the meantime, blind people will have
> difficulties using it. So, once you satisfied the accessibility
> standards you have picked, depending on your page, it might be a good
> idea to ensure that it works with a screen reader.

Actually, putting a screen reader in the hands of somebody who rarely uses a
screen reader will do little to actually verify the accessibility of a
document.

Screen readers are complex and sophisticated pieces of software that require
almost daily use to master, and most web developers cannot devote the time
necessary to properly learn how to use a tool such as JAWS: it's really more
than just ready your page out loud. You need to understand the various ways
that users of the tool (who themselves have varying degrees of competence
with the tool) can navigate around the screen / web page. You need to
understand about verbosity settings, how the tools can be personalized for
individuals (which may or may not affect outcomes), the various keystroke
combinations used to interact with the tool
[http://www.wats.ca/show.php?contentid=48] and much, much more. Finally,
just because your page "works" in JAWS is no more a guarantee that it is
"accessible" then saying that it also works in Internet Explorer. In short,
save your money and don't buy JAWS unless you plan on using it every day.

A much better methodology is to actually find a daily user of adaptive
technology and work with that person to do human testing of your site. Sit
with that person and have them review the site in question. Ask questions,
try to achieve "results", but avoid directing the user on how to do
anything... Sighted testers using JAWS is not a real test: you can see
things and whether intentional or not, your "seeing" will skew the results.
Instead, have the non-sighted user go through the site and verbalize their
"journey". What's good for them? What's bad for them? Set a task and ask
if they can achieve it. Never mind "how", the real question is can they,
and can they do so easily? Ask them to summarize the key points on the
page. Does the summary accurately reflect the visualization? Often large
areas of screen real estate is dedicated to something "important", yet
semantically the content is relegated to just another paragraph. In short,
try to glean from the user their impression and usability of the page. If
they can achieve everything that the sighted user can do, *then* you can
safely conclude that the page is functionally accessible. (This may also
contradict some technical measurements, at which point you as the "expert"
need to weigh the pros and cons and report back to the content owner - for
example if a page does not validate, it will not pass WCAG AA - even though
the page might be totally accessible to all of your test subjects.)

I realize that for most developers this level of testing is quite extensive,
and probably beyond reach more often than not. That's really too bad (and
I'm being sincere here), because at many levels it is this type of testing
that should be done, but often is not. But if you *really* want to get a
better understanding of the issue, you *really* need to go though the above
exercise at least once in your career. Seek out volunteers (or trade
services) - most communities have support groups for their constituents (and
remember that web accessibility is more than just visual disability).

So to summarize, and return to the question initially posed: to truly
determine accessibility *does* require human testing, but by humans who
access web content using the various AT tools out there, not by web
developers buying a piece of software and attempting to replicate foreign
experiences.

Cheers!
JF
==================================John Foliot
Academic Technology Consultant
Stanford Online Accessibility Program
http://soap.stanford.edu
Stanford University
Tel: 650-862-4603
==================================

From: Keith Parks
Date: Fri, Mar 14 2008 9:20AM
Subject: Re: accessibility without testing?
← Previous message | Next message →

On Mar 13, 2008, at 4:26 PM, John Foliot - Stanford Online
Accessibility Program wrote:

> [snip...]
>
> I realize that for most developers this level of testing is quite
> extensive,
> and probably beyond reach more often than not. [snip...]
>
> So to summarize, and return to the question initially posed: to truly
> determine accessibility *does* require human testing, but by humans
> who
> access web content using the various AT tools out there, not by web
> developers buying a piece of software and attempting to replicate
> foreign
> experiences.

So given all the difficulties and variables in human testing you
point out, doesn't it realistically come down to designing to
standards (checked with automated and manual inspection), combined
with proper document information structure (html mark-up)?

If you do those things, it would seem like you'd be, say, 99%
covered. And to uncover that last 1% of something that may not be
accessible in a site, you'd have to do multiple human tests, with
users having various kinds of disabilities and skill/experience
levels, on every page of a site. And in *my* real world at least, the
level of real disabled user testing I might be able to arrange
doesn't seem worth it on a regular basis. (Though I definitely agree
that it's a powerful lesson to sit with a real disabled user as they
experience the Web with a screen reader, or try and navigate a
complex page without a mouse, etc.)

In other words, if you have a long enough list of checkpoints, there
shouldn't be a need for human testing, since nothing short of full,
systematic human testing is going to uncover problems that can't be
uncovered some other way.

******************************
Keith Parks
Graphic Designer/Web Designer
Student Affairs Communications Services
San Diego State University
San Diego, CA 92182-7444
(619) 594-1046
mailto: = EMAIL ADDRESS REMOVED =
http://www.sa.sdsu.edu/communications

http://kparks.deviantart.com/gallery
----------------------------------------------------------

World Peace through Cascading Style Sheets.

From: Steve Green
Date: Fri, Mar 14 2008 1:20PM
Subject: Re: accessibility without testing?
← Previous message | Next message →

If you don't do user testing and you don't engage accessibility consultants
and you don't spend time with disabled people to learn about their
disabilities, then there is a very significant possibility you're going to
build something that is inaccessible.

The WCAG are a starting point but how close they get you to an accessible
site will vary greatly depending on the content, the functionality and the
technologies you choose to build it. In some cases designing to standards
may be good enough. In other cases it may be totally inadequate. How are you
going to know which it is?

A long list of checkpoints isn't the answer. A lot of the problems users
have are not technical, but cognitive; they can access the content but can't
understand it or can't understand what they need to do. Often this is a 100%
blocker, not 'the last 1%'. With our experience of user testing we can
assess whether users are likely to find a website accessible, but user
testing frequently reveals unexpected issues.

The legislative environment in which you work will influence your approach.
In the UK we have the Disability Discrimination Act (DDA), which applies to
websites. This does not specify that websites must meet any particular level
of technical accessibility (such as WCAG). Instead it is interested only in
actual outcomes i.e. whether people can actually use the website. This is in
direct contrast with Section 508, which was specifically drafted such that a
totally objective technical pass/fail assessment could be made. Although the
DDA is far more subjective, it meets users' needs better than Section 508.

Steve

From: Keith Parks
Date: Fri, Mar 14 2008 2:30PM
Subject: Re: accessibility without testing?
← Previous message | Next message →

On Mar 14, 2008, at 12:08 PM, Steve Green wrote:

> If you don't do user testing and you don't engage accessibility
> consultants
> and you don't spend time with disabled people to learn about their
> disabilities, then there is a very significant possibility you're
> going to
> build something that is inaccessible.

I'm not urging people not to engage disability consultants or not to
spend time learning first hand from disabled people how they
experience the Web. I was only talking about user testing.

> The WCAG are a starting point but how close they get you to an
> accessible
> site will vary greatly depending on the content, the functionality
> and the
> technologies you choose to build it. In some cases designing to
> standards
> may be good enough. In other cases it may be totally inadequate.
> How are you
> going to know which it is?


> A long list of checkpoints isn't the answer. A lot of the problems
> users
> have are not technical, but cognitive; they can access the content
> but can't
> understand it or can't understand what they need to do.

I'm truly and sincerely curious as to what types of issues people
have discovered through user testing that had not been noticed during
development and manual inspection of a site? Are they all content
issues?

It just sounds like the types of issues that would come to the
surface that way would be so... individualized; a particular user
having a problem comprehending a particular piece of content. And
would these issues apply to non-disabled users as well? (We've all
experienced the case of the developer being too close to the content,
but those types of things ought to be caught in the regular non-
disabled, third party review process.)

And since we can't realistically test *all* types of users (are there
"typical" types of cognitive disabilities?), it just seems to me that
you would hit the point of diminishing returns pretty quickly with
anything but an extremely wide range of user tests. An "average"
disabled user test would only catch the low hanging fruit, so to
speak. Which designing to standards and "best practices" should catch
before they happen.

It almost turns into a Rumsfeldian dilema: There's the known, and
then there's the unknown. And there are the unknowns that are
knowable, and the unknowns that are unknowable.

Building to the standards and thoroughly inspecting the results
should cover the "known".

To uncover the "unknowable unknowns", you'd have to test a big enough
variety to reasonably represent *all* users. Clearly not practical.

So it comes down to testing for the "knowable unknowns". And is
having typical (assuming there is agreement as to who that is)
disabled users test sites the only way to uncover those? I'm not
convinced it is.

But again, I'm curious to hear about specific problems that testing
has uncovered that were otherwise "unknowable."

Thanks for listening,

Keith

******************************
Keith Parks
Graphic Designer/Web Designer
Student Affairs Communications Services
San Diego State University
San Diego, CA 92182-7444
(619) 594-1046
mailto: = EMAIL ADDRESS REMOVED =
http://www.sa.sdsu.edu/communications

http://kparks.deviantart.com/gallery
----------------------------------------------------------

A riddle wrapped in a mystery inside an enigma, served with a side of
slaw.

From: Bob Cavanaugh
Date: Sat, Mar 29 2008 3:40PM
Subject: test
← Previous message | No next message

I'm new, test.
From,
Bobby Cavanaugh
Email: = EMAIL ADDRESS REMOVED =
Cell: 206-418-8763