WebAIM - Web Accessibility In Mind

E-mail List Archives

Re: Web application testing

for

From: Tim Harshbarger
Date: Aug 8, 2016 7:22AM

There is not a single answer to this question. You will probably get different answers from different people. Each of those answers are likely good ones, but may only best apply to their situation.

The right answer for you is going to involve balancing the results you want to achieve with the cost and time it is going to add to implementing the application.

Every browser and screen reader combination you add to your testing increases the time it takes to complete the work and the number of potential defects you will need to work through. Not to mention that the people testing with the screen reader will need to be somewhat knowledgeable about the screen reader.

And as you might have noticed, people are also mentioning testing browser/screen reader combinations using different versions of screen readers and browsers. And that increases the complexity of the question.

You really need to start by deciding what your goal for accessibility testing is and estimate how much time and cost can you spend on that piece. Is your application available both for desktop and mobile? Which browsers do people most frequently use to access the application?

Who is your audience? Which screen readers are likely to be available to them? In a closed environment, everyone might be using just one or two screen readers. In an open environment, which screen readers might be used most frequently?

Is your goal to conform with WCAG 2.0? Then maybe you want to consider which browsers and screen readers provide you the best opportunity to do that without a lot of hacks.

Perhaps, you decide just to start small with a single browser and screen reader--and look at adding others over time.

There are a couple pieces of advice I would personally add...

Start with the assumption that you will never have an application that is 100% free of accessibility defects. During every iteration of the application, you need to aim for fixing whatever accessibility defects you can fix for that iteration. Know that some defects are going to get through. But your plan is to catch them in the next iteration. You will ultimately find that as technology and techniques change, you will find new defects or realize there are now better ways to fix old defects.

The second piece of advice I have actually comes from my father who use to be a lead tester for a large complex system for a few decades. The key purpose of testing is not to let any of the big or nasty defects through. As long as your testing can catch those types of defects and you can ensure they are fixed before your users has to deal with them, you are doing good. Repairing more defects than that would be even better. Repairing all the defects would be grand. But catching and stopping the defects that prevent people from being able to use your app is essential.

If you are just starting your testing, chances are you will find a lot of those "big" defects and you will want to spend most of the effort clearing those out. Over time, you might see less of those and find yourself with more time to go after other less critical but still important defects.

Thanks,
Tim