In the earlier years of web accessibility (pre-1999 to 2004), screen readers were buggy and temperamental. They would sometimes respond erratically to even simple content. Several of the out-dated recommendations of WCAG 1.0 were put in place to address these screen reader-specific issues (see almost any checkpoint that starts with “Until user agents…”), but problems remained and screen reader testing was often a necessity, even for relatively basic pages.
Since then, screen readers have improved. They are far from perfect, but they have fewer bugs and tend to reliably read correctly-structured content. If you build a simple web page following solid accessibility guidelines, you can be relatively confident that your page would be accessible to a screen reader without actually testing it with a screen reader. I think this was one of the main reasons that, between around 2004 and 2007, I relied less on screen reader testing as part of my accessibility evaluations. It was still important in some cases, but many pages could be evaluated using a good evaluation tool like WAVE (provided by WebAIM) and by performing a few other manual checks, such as testing keyboard accessibility.
The web is much more dynamic and interactive than it was just a few years ago. Some of these changes have promising implications for accessibility, but they also introduce new accessibility challenges. Highly dynamic content, which is becoming more common, needs to be tested in a screen reader. This means screen reader testing as part of a larger accessibility evaluation is becoming more common as well. Today, it is a key part of almost all my accessibility evaluation work. Not only is my screen reader testing more frequent, it is more focused. In addition to listening to a page being read from start to finish, I usually concentrate my testing on certain areas, such as forms and navigation.
I have heard, and likely said, that testing with a screen reader very difficult. That it takes weeks or months to learn even the basics. I don’t agree with this statement anymore. Basic screen reader testing really only requires familiarity with a few simple keyboard commands and an understanding of what to listen for. Resources are available (e.g., our articles on testing with JAWS and testing with NVDA) to help users learn the basic shortcuts. In my opinion, the knowledge required to implement accessibility is much more difficult to acquire than knowledge of how to actually use the screen reader. If anyone feels like screen reader testing is an unnecessary or overly difficult part of accessibility testing, I hope you give it another chance.
Things have improved, but I’m not convinced that testing with a screen reader should be encouraged for developers in general.
It’s too easy to make decisions based on limited experience that don’t match how people who need to use a screen reader, would actually use it.
Developers with limited experience (including me before 2003) would make changes specific to how a particular screen reader works (e.g. JAWs) or specific to how they hear it (e.g. phonetic spelling).
I would like to encourage developers / testers to use screen readers, but there needs to be some method of understanding what ‘normal’ usage is.
Until you use a screen reader like JAWS on a regular basis as I do, you cant understand the problems that we encounter, problems as simple as switching too fast between windows, trying to activate JAWS while the page loads only to have it crash, even this form presented problems in filling it out.
Simply testing your page under perfect conditions does not give you the true picture and can cause even more problems in the wrong hands. A webmaster working for a company or government who tested with a screen reader tells their Boss that it is screen reader friendly and anyone who dares email them and say differently is dismissed because said webmaster is now considered an expert
Interesting perspective Jon, but I have to agree with Alastair and Geof.
I am always concerned that a developer will pick up a screen reader fiddle about for 5 minutes and be convinced that they are able to use it well. This is not the case in my experience. Yes screen readers are easier than they were, however there is functionality in there still that a newbie would not find and some of it can even facilitiate access around more complicated software and web applications.
Yes it is great if developers take the initiative and yes they should be encouraged to find out what it is like to navigate around the screen using a screen reader and using screen magnification. But I still think that more formal testing should be done whilst being supported by someone who really does know about screen reader behaviour.
I have worked with developers who have got excited by JAWS, which is great but they have focussed on JAWS to the exclusion of everything else and actually have hampered accessibility more than they have improved it.
After all we all know that something being read by a screen reader does not mean that it is ‘accessible’. There is still a broader accessibility picture and it is not just about being accessible with JAWS.
What are the most effective ways for non-disabled web designers and developers to get truly meaningful testing with products such as JAWS?
Usability tests with blind/visually-impaired users of these assistive technologies? Working with local organizations that provide such equipment?
I’m still semi-new to the game and have been wondering the best way. I definitely can see the point that developers using screen readers to test their sites isn’t quite the same experience as someone who uses them out of necessity navigating through the entire site.
Either way, interesting article, comments included!
Thank you for your excellent comments so far. AlastairC, Geof, and Sally all make great points. I think the need for screen reader testing by experienced testers, and true screen reader users, is an essential part of a good accessibility evaluation. We always make this recommendation as part of our trainings and I should have made this clearer in the post. Allow me to clarify. If you are new to screen readers (and accessibility in general), I recommend that you seek feedback from experts *in addition* to your doing your own testing. If you’re not sure about something, be sure to seek advice from more experienced screen reader users/accessibility advocates.
Rookie screen reader users may make mistakes that can negatively impact accessibility, but I think testing with a screen reader is more valuable than not testing with a screen reader for fear of making mistakes. Saying “leave it to the experts” sort of runs contrary to WebAIM’s core mission.
I’m afraid the last paragraph of this post sort of overshadowed its central theme: an accessibility evaluation of dynamic content is incomplete if it does not include screen reader testing, and this dynamic content is becoming a fundamental part of today’s web.
A few targeted replies:
AlastairC, you mention making a few rookie mistakes yourself in 2003, but now you recognize yourself as an experienced screen reader user. Could you have reached your current level of expertise without making some of those mistakes along the way? You also identify the need for “some method of understanding what ‘normal’ usage is.” You are absolutely right. WebAIM is currently working on an article to address that very need. We hope it will help new screen reader users avoid some common pitfalls and also offer some tips on how to make their screen reader testing more effective.
Geof, First off, apologies for the accessibility gaffe on our own comment submission form. We must have missed that while updating WordPress. It has been fixed. Interestingly enough, it’s the perfect example of the kind of thing that would have been uncovered with a bit of screen reader testing, which I guess is our second gaffe.
The example in your last paragraph is valid; however, it would be equally valid if you replaced “screen reader” with “accessibility,” “usability,” “standards,” or any number of things.
Sally, you make a great point about relying too heavily on screen readers at the exclusion of greater accessibility. At the last minute, I pulled a sentence very similar to this out of my post. Bad Call 🙂
Steve-
As I mentioned in my reply to AlastairC, we are currently creating an article that should offer some pointers on screen reader testing. Summer is a pretty hectic time for us, so it might be a couple of months.
Usability testing with users with disabilities is an excellent idea and local organizations are often a great resource. If you’re having a hard time finding volunteers, you might want to try our email discussion list. Someone might be able to connect you with a willing helper.
I took a quick look at your blog. You won’t be able to call yourself semi-new for long!
Well, I felt a bit better after reading Steve’s comment and John’s reply. I strongly believe that one should always test their website with a screenreader as part of an accessibility check, and I’m sure there’s a good number of folks out there who’d like to help that don’t need any training to do it. Web developers don’t need to learn JFW or Win. eyes, (unless they want to that is), they need to have one of the thousands of screenreader users already out there test it for them. Not only would I, (and some others I know), be happy to volunteer, considering I kind of like to be able to use the internets and such, but getting paid would be even nicer. The blind screenreader user doesn’t need any training, and will be able to test much more effectively for accessibility. It’s hard enough for blind people to get jobs, (over 70 percent are unemployed), and having one of the people you’re making your page accessible for tell you if it is makes the most sense and will help developers get it right the first time.
Hi I have read all your posts and comments with interest.
I work with young people with physical impairments to support them through transition from childrens to adult services and one of our projects is to devise a transitions website.
We are currently looking at making it as inclusive and accessible as possible so have been investigating screen readers, which are a new concept for me. If we were to use one to test our site how would we go about this? Some of our clients do have visual impairments whilst others may be challenged educationally by the complexity of their disability and simply be physically unable to read.
I’m sorry if I am high-jacking your discussion by changing its focus slightly but you are all obviuosly experts in this field and I am not sure who else to ask.
I tried NVDA and got into it after a day, there was a list of navigation commands in the help file. What was more difficult though was figuring out what part of the document I was in. Let’s hope ARIA will make things easier in terms of ajax applications…
I believe that one should always test their website with a screenreader as part of an accessibility check, and I’m sure there’s a good number of folks out there who’d like to help that don’t need any training to do it. And the Web developers don’t need to learn JFW or Win. Really a nice post…
Having a site like this and http://www.macrotesting.com makes me to clear my doubts and to improve my knowledge in a well manner. Thank you…
cheers
sakthi
It is unfair to say that some years ago screen readers were buggy and temperamental. They were and are great assistive technology tools that permit computer access and make the best of a bad situation. Even today content developers write / generate (via tools) and invalid HTML code that do not conform to HTML specs, does not expose structure etc. When form controls do not have labels or data tables do not have their header cells marked up, screen readers try to convey the data-relations as best as they can so that users may still be able to access content. In doing so screen reader developers made different assumptions and adopted different frameworks for their products. They differ in their level of support for different attributes too. That is why user-experience with different screen readers is different. And as technologies and browsers have evolved, so have screen readers.
Browsers too have their own idiosyncrasies, right?
And the clause ‘until user agents…’ in WCAG 1.0 applies to both browsers and assistive technologies.
Regards,
Sailesh
Hi all,
I am currently working on my thesis and a friend of mine who is blind gave me this site to gain some information. I thought I might see if I could enlist anyone’s help from here. My thesis question is “Would giving the disabled community better computer and Internet access have an impact on the economy?”
Gretchen Maune: you stated that 70 percent of the blind are unemployed. I would love to know where you obtained that statistic.
I am trying to get some data on what costs are like to make accessible websites (new websites and making existing websites accessible). If anyone knows where I can get that information, it would be greatly appreciated.
I am not concentrating solely on vision impairment or blindness, but, I do believe, and please correct me if I am wrong, that vision disabilities are one of the most difficult disabilities to assist when it comes to making computers and the Internet more accessible.
In a former job I assisted with setting up a Section 508 lab and I became very interested, hence my thesis topic. It is truly difficult to adequately test with screen readers when you are sighted and we did hire a blind person for a summer to give us some insight into what she had to contend with when using computers and the Internet.
I am also about to send out a survey that will be targeted to the disabled (any disability will fit the eligibility criteria, as long as they are US citizens and need to use assistive technology to use computers). Getting a large enough target audience is going to be a bit difficult for me, so, if any of you have any ideas on how I can get this distributed to a wider audience I would be grateful.
Thanks in advance for any help anyone can provide me.
Sondra
Sondra,
Hi, sorry for the delay, I just read your post. The stat I was/am using is for the US, and about ten years old, unfortunately, since I think It’s from the last census, but here is a quick link to the stat, and one can find many others doing a google search for blind and unemployment.
http://findarticles.com/p/articles/mi_qn4182/is_19990706/ai_n10129809/
Thanks Gretchen.
If anyone out there is willing to take my Thesis disability survey, here is the link, and please feel free to pass it on to as many people as you can!! THANKS ALL!!
http://www.surveymonkey.com/s.aspx?sm=SfKo9B_2fPPp_2bu_2f2Hc0i4M7A_3d_3d
Bear in mind that screen readers, such as JAWS, are web accessibility applications. Screen readers are *not* web accessibility testing tools. A screen reader cannot test the validity of a web page and its support for Section 508 nor WCAG 1.0/20 because testing is not a designed function of the application. In other words, if all you have is a hammer, everything you see is a nail.
Accessibility testing is more complex than that, requiring a combination of real accessibility testing tools, auto-testing, manual testing, human testing, etc. And that testing has to be objective, logical, documented and transparent. Documenting the testing process is very important so that on any given day, any two more given testers using the exact same tools and following the same instructions should identify the same accessibility issues from the same page. Their evaluations should be practically identical.
However, without such transparency, restricting testing to just screen readers (or even placing too much emphasis alone on screen readers for testing) will result in less than honest evaluations. Couple that with political pressure and the evaluations results may turn out to fit what the powers that be want, and not the honest reality.
Take a look at:
http://www.508portal.com/?q=node/630
http://css-discuss.incutio.com/?page=ScreenreaderVisibility
There are many different kinds of accessibility testing; some will require screen reader testing more than others. In some cases the target site or application and its universe of users should be factored into the screen reader testing protocol. For example, an e-government site should be accessible to and usable by anyone with any screen reader at any level of expertise. At the other extreme, someone testing an internal site only used by sysadmins may be able to assume greater expertise and fewer screen readers.
Do you need to use tables for the layout of your website.
Screen reader-based accessibility testing is fine, if you want to know if there is a specific issue between the content you are testing and a specific screen reader and version of that screen reader. Beyond that, its not easy for general IT developers to do, is time consuming in light of other visual and automated review techniques available today, and is fraught with versioning issues, and issues where one screen reader is able to access content because that particular product has heuristic methods to overcome inherent inaccessibility. Because screen readers in general do overcome inaccessibility, often in the background, they don’t give you the real picture if the content “should” work or not, they only tell you if they work for this combination of products. A more general solution is important, because when you know if the content “should” work, you can focus remediation efforts to a screen reader vendor or developer if needed, or back to a content developer as the case may be. Depending upon screen readers for testing also skews the perspective of testers as to the set of people with disabilities and can cause oversight of other issues. Also, screen readers don’t do well at many testing tasks by their nature–e.g. you can’t quickly test for use of color with a screen reader, determine time out with a screen reader, or in some cases even reliablly determine if keyboard accessibility is complete. All in all, I think to be very sure that a site is accessible you might employ a few screen readers to “know” if any issues are out there, but if your site is coded to W3C WCAG, or Section 508 standards, and you’ve validate that, you’ve done what you should be doing, what’s left is the AT vendor/developers responsibility.
Well as far as I am concerned this text box is about the most inaccessible widget that I’ve come accross for a long time. The text keeps disappearing behind the right borders – so I cannot see what I am typeing. Mind you this wouldn’t trouble a blind user.
I think the key things are :
Define standards for writing code that is accessible. My experience here is that exposing developers to code that doesn’t follow those standards in conjunction with a screen reader (or magnifier etc) can really help a developer understand why those standards are important in developing an accessible website / application.
Do not make the mistake of thinking that because a JAWS user can access information, that that is job done. This text box is a prime example!!! (I’m still on IE6 – so may be a browser issue!)
Any developer can test how the application behaves when not using a mouse. Testing with the keyboard alone is probably the single most important testing for accessibility that ANY developer can do.
If you have standards, then it is inevitable that they will get stretched over time. In particular developers will want to introduce new widgets / controls etc that may or may not have been writtenn with accessibility in mind (JQuery anyone — though good to see that they are implementing aria)
W3C standards are OK as far as they go, but they are better suited to presentational web pages rather than applications delivered via the web. Putting a tabindex=”0″ on a can be of great use in aiding accessibility but is not W3C complient, fo instance (I wait for the flamethrowers for suggesting that one). Access Keys are depreciated in HTML5 — but for commonly used buttons within an application they can be a real time saver for users who cannot use a mouse. (Ever got fed up tabbing through a list of records to be able to press a button at the end of the list!).
I agree with most of the points that people have made. The one thing that can be said is that slowly using screen readers has become generally easier and I think an average Web developer could now become familiar with Screen Reader usage and the web in a day or so — whilst still recognising that he may well miss some accessibility issues that a blind or low vision user might find highly annoying.