WebAIM - Web Accessibility In Mind

E-mail List Archives

Re: DHS Trusted Tester_Testing support/input

for

From: Steve Green
Date: Aug 29, 2024 9:26AM

You have raised a lot of different issues, so here are my thoughts on some of them:



* As far as I can tell, Captivate courses are published as HTML5. I cannot see any justification for “testing them as software”, whatever is even meant by that. And most software accessibility standards require pretty much the same as WCAG except where the success criteria are not relevant to an application not written in HTML. Just putting a SCORM wrapper around a web-based learning module doesn’t magically turn it into something else.

* You are correct with regard to headings. Tools identify programmatic heading, not visual ones. It’s the auditor’s job to identify visually apparent headings and verify that they are also conveyed programmatically. I would do that by inspection of the source code rather than using ANDI, but that’s a different topic.

* I really, really don’t like ANDI and never use it. We use a huge array of other tools, predominantly single-purpose bookmarklets, but also the Deque Axe and ARC Toolkit browser extensions. For testing a whole website, we use SortSite.

However, we only use the tools to support our manual testing. The pass / fail decision must always be based on inspection of the source code and user interface. You never trust what a tool tells you.

* This last point feeds into the wider observation that the Trusted Tester methodology is fatally flawed. It was devised to avoid the inconsistent results that invariably occur when different people test the same website. The strict methodology, which mandates using ANDI, ensures high repeatability, but at the cost of accuracy. This is a poor trade-off in my opinion.

We can train skilled testers so their results become more consistent, but the Trusted Tester methodology will always be inaccurate. In fact, it will probably become more inaccurate as coding techniques change and the methodology is not adapted to them. FWIW, I would refuse to ever use the Trusted Tester methodology because it forces you to do bad work.

Steve Green
Managing Director
Test Partners Ltd