WebAIM - Web Accessibility In Mind

E-mail List Archives

Re: [EXTERNAL] Graphical element instead of the role link presence. How should I report it?

for

From: Mark Magennis
Date: Mar 4, 2021 5:33AM

Thanks for injecting this Steve. I think it's very useful to discuss approaches to testing and it does seem relevant to this discussion.

Despite having said the polar opposite, I understand your point and yes, if starting with an automated tool does negatively affect the quality of a subsequent manual testing then that is a problem. I haven't observed this in practice but that may be because I haven't sufficiently studied it closely enough.

However, for most of those in my company who do accessibility testing, I think starting with aXe has a lot of benefits. All members of the development teams - UX, Engineering, and QA - are expected to cover accessibility, but in practice this is only to the extent that they are able. We use automated testing as the minimum and expect/hope some degree of manual testing will follow on from there and that's where we run into limitations. Engineers don't have time to do much testing but we ask them to at least run aXe and fix Violations before handing over to QA. We also ask them to try the functionality with the keyboard only and check what the screen reader reads, though I know a lot of them don't do that. Even if they do, we run into the problem where they don't really understand how it *should* work with the keyboard and what a screen reader *should* read, so their manual testing is not always effective. QA are expected to do much more in depth testing, as is normal in their role, and hopefully develop accessibility knowledge over time. We still recommend they start with aXe, if only to check that there aren't any Violations that Engineers should already have found and fixed. But with manual testing, QA run into the same problems as Engineers - not having sufficient understanding of how something should work for it to be considered accessible. They have a process and a lot of supports (defined testing process, a11y user acceptance criteria, learning resources, etc.), but they often still have insufficient time to learn and put their learning into practice. If they were able to use all of the resources and do the full manual QA check to a high degree of expertise, then there would be little need for them to use an automated tool. But in practice, some QA Engineers are entirely new to accessibility, most have very little time to devote to a11y QA (let's face it, the overhead is enormous), some squads or product owners are just not sufficiently bought into it to allocate the time, and some QA staff just don't get it or don't have the level of interest for it to spark their curiosity and wish to learn more. Very few have ever sat down with real users with disabilities or observed user testing for example, so their understanding of what we are trying to achieve is often very limited. We do use offshore QA who have a lot more a11y experience and they can follow the process much deeper, but they are still limited in their understanding. The specialists in the Accessibility Office are the ultimate testing and knowledge resource but we are painfully few and very stretched.

Given this, we find that getting the dev teams to at least use the automated tool and think about what it is telling them is a very good start and often as far as we will get. But it does give a good platform for training/mentoring discussions. The Accessibility Office runs regular a11y workshops for UX, Engineering, and QA and the teams often come with questions around "aXe is saying this but I don't fully understand it". I find that's a really good start for conversations that will help to upskill them.

The Accessibility Office can do a complete manual audit without needing automated tools, but as you say it's useful as a check to see if anything was missed. However, I personally always start by running aXe and seeing what it says. I don't really know why I start with this. I didn't used to but I've grown to like it. Maybe it just feels like it gets me going easily. I don't feel it affects the manual testing I do and yes, it does misdiagnose a fair amount but I can understand that. So for specialists maybe it doesn't matter but for people strapped for time or with little knowledge then I feel it can be useful to start with automated tools.

I think I'll pay more attention in future to looking to see whether it might hamper subsequent manual testing though because I can see your point.

Best,
Mark

-----Original Message-----
From: WebAIM-Forum < <EMAIL REMOVED> > On Behalf Of Steve Green
Sent: 04 March 2021 11:25
To: WebAIM Discussion List < <EMAIL REMOVED> >
Subject: Re: [WebAIM] [EXTERNAL] Graphical element instead of the role link presence. How should I report it?

Although it's not strictly relevant to this discussion, I strongly recommend the exact opposite. In most contexts, I advocate conducting a full manual test first, and only then use an automated tool as a safety net.

The rationale, which I have observed in practice, is that the automated tool results adversely affect how people do the manual testing. They miss issues the tool didn't report, perhaps because there is a subliminal voice telling them there's nothing wrong there. They tend to report issues exactly as the tool reported them, even if the tool misdiagnosed them, which is not uncommon. This shouldn't happen, but it does.

Steve Green
Managing Director
Test Partners Ltd


-----Original Message-----
From: WebAIM-Forum < <EMAIL REMOVED> > On Behalf Of Mark Magennis
Sent: 04 March 2021 11:13
To: WebAIM Discussion List < <EMAIL REMOVED> >
Subject: Re: [WebAIM] [EXTERNAL] Graphical element instead of the role link presence. How should I report it?

You're welcome Vsevolod.

I should add, I think it is always useful to begin any review by running an automated accessibility testing tool on the page. It will catch quite a few issues (somewhere around a quarter of them in my experience) and flag some suspected issues. I use aXe from Deque. You can download it for free and run it from the F12 inspector in the browser. It is simple to use and produces very clear results. It shows you the things that it knows are errors and points you to some other things that it suspects may be errors that you can check manually. It's a great start. It also links to further information and explanations which can be very useful in developing your knowledge of accessibility.

Mark

-----Original Message-----
From: Vsevolod Popov < <EMAIL REMOVED> >
Sent: 03 March 2021 18:45
To: WebAIM Discussion List < <EMAIL REMOVED> >; Mark Magennis < <EMAIL REMOVED> >
Subject: Re: [WebAIM] [EXTERNAL] Graphical element instead of the role link presence. How should I report it?

Hello Mark,

Thank you for your answer!

Yes, the first thing that I reported was the inability to navigate using the keyboard with tab key and also I reported about the menu which doesn't have link elements and only has the div elements.

Thank you for your tips,

I will surely use them when doing my next reports!

Thank you again!

03.03.2021 19:45, Mark Magennis пишет:
> Wow!
>
> That site is really terrible from an accessibility point of view. The logo Home page link is really the least of the problems.
>
> The biggest problem is that the site as a whole is completely inaccessible using the keyboard because you cannot Tab to any of the interactive elements.
>
> It's slightly better using a screen reader because at least you can arrow through it, read most of the content, and activate many of the menu items.
>
> I would suggest you mention these two points and then arrow though it with a screen reader yourself to find other issues you can point out in your report, such as:
>
> 1. Some content, including navigation menu items, is skipped.
> 2. Navigation menus don't behave correctly (e.g. you can't close them
> without selecting an option) 3. Some headings are not marked up and heading levels aren't correct.
> 4. Information is contained in images without text alternatives.
> 5. Some non-informational images have unnecessary and meaningless text alternatives.
> 6. Etc.
>
> You can look at a lot of the HTML code in the Inspector - right click on any element and select Inspect from the menu.
>
> Good luck,
> Mark
>
>
> --
> Best regards,
> Vsevolod