Making VPATs and ACRs More Effective in Procurement

Name of Product/Version and Report Date

These give us insight into whether the ACR is current and applies to the software version we seek to purchase. When cloud-based platforms don't have a version number per se, the report's date becomes much more important. Ideally, a vendor provides an ACR that is within a few months to a year of the product purchase date and/or that applies to a release version that is at least the same major version number as the product under consideration. Otherwise, the purchaser must ask "What has changed in the product since this ACR was created that has a material impact on accessibility?" This is one area where less technical staff can begin vetting the ACR, potentially making the process more efficient.

Contact Information

Having a contact that is from UX design, web development, or accessibility is critical to collecting more technical information if required. Some vendors will provide contact information for a sales or product role because they always place someone between potential clients and more technical staff. When this is the case, purchasing organizations should make sure that expectations are clear regarding the expected turnaround for responses, as well as the nature of the roles that the vendor has in place to respond.

Notes

The instructions for vendors in the VPAT indicate that vendors can provide "Any details or further explanation about the product or the report. This section may be left blank." The VPAT, "Best Practices for Authors," suggests more (Information Technology Industry Council, 2022):

• Additional information about the product version that the document references
• Any revisions to the document
• Links to any related documents
• Additional information describing the product
• Additional information about what the document does or does not cover
• Etc.

This is a section where purchasing organizations can discover more about the candidate vendor, the candidate product, and the vendor's accessibility practice. The purchasing organization may require vendors to provide insight into how they integrate accessibility into product management and design in Notes, often making the request in an RFP. This is the only space in the VPAT where a vendor can provide this kind of information.

Knowing that a candidate vendor has a more mature internal accessibility program gives the purchasing organization more peace of mind. It is one thing for a current version of a product to be strong in accessibility relative to other products in the candidate pool. Knowing that a candidate vendor has made accessibility a part of its operations upstream from quality assurance and even development suggests that the product will remain more accessible over time.

An indication that accessibility is a part of a vendor's core mission and values, including stated commitment and vision from senior leadership, is ideal. A sign that vendors have at least embraced accessibility at the product level, including insight into professional development provided, UX design practice that includes accessibility, and similar demonstrates at least a more holistic and desirable approach.

Evaluation Methods Used

The VPAT's instructions for vendors are a bit narrow, stating, "Include a description of evaluation methods used to complete the VPAT for the product under test." Best practices suggested in the VPAT include testing based on knowledge of general product functionality, testing with assistive technologies, published test methods, a vendor proprietary test method, or other test methods.

This opens the door for a vendor to enter "General product knowledge" or similar, which is relatively common. When purchasing organizations see this claim in isolation, they should ask for more specific information. This is true when the response does not provide detailed insight.

An ideal response from a vendor will include details such as tools and processes used to evaluate the product. When specific tools are referenced, they should be referenced by name, whether "WAVE" or the name of a particular assistive technology. Testing using only a keyboard is also a positive indicator.

Insight into who performs testing is also a key focus area. Vendors that indicate that they have engaged people with disabilities in testing may be further ahead of other candidate vendors.

Technical Details

This is often the primary focus for purchasing organizations, with good reason. The VPAT allows candidate vendors to self-assess the accessibility of a product using different accessibility standards. Regardless of the standard set used by the purchasing organization, a high level of scrutiny of the information about conformance to standards reveals needed information without dedicating as much time as direct accessibility evaluation.

For WCAG 2.2, for example, the VPAT places each of the WCAG Success Criteria into a table row. The name of each Success Criteria is in the first cell of the row. The vendor then states how well they meet the requirement in the second cell using one of the following terms:

• Supports: The functionality of the product has at least one method that meets the criterion without known defects or meets with equivalent facilitation.
• Partially Supports: Some functionality of the product does not meet the criterion.
• Does Not Support: The majority of product functionality does not meet the criterion.
• Not Applicable: The criterion is not relevant to the product.
• Not Evaluated: The product has not been evaluated against the criterion.

Then, vendors are prompted to provide "Remarks and Explanations" in the last cell in the row. This cell cannot be empty, even if the vendor states a conformance level of "Supports." All this information is vital to understanding the claimed level of conformance and discovering where there are gaps in accessibility and even when to expect those gaps to be filled.

We will examine common responses, including red flags and opportunities to relay specific questions to vendors.

Scoring

Having a scoring system is helpful when deciding on technology purchases. This helps to bring more consistency to the process, particularly when the scoring system considers all variables considered during decision-making.

Including accessibility in a scoring system ensures it is considered alongside functional and business requirements. This is an excellent way to normalize accessibility as a critical component in the decision-making process. It also helps to ensure that a product owner cannot push a purchase through when the product is demonstrably less accessible than others in the product pool.

The management of scoring an ACR also ties into organizational capacity. More nuanced or complex scoring strategies require more in-depth accessibility knowledge and more effort. However, this may not be to the detriment of a more intensive scoring system relative to the time spent performing direct testing.

Scoring for elements of the vendor information (Notes and Evaluation Methods Used) and for each technical Criteria may consist of assigning a score to each item in the ACR. A score of 0 means that the vendor did not provide needed information, that the information is not helpful, or for technical criteria, that the Criteria is not supported at all. Points may increase as information on the ACR indicates more helpful information and disclosure of indicators of a more mature accessibility program. For technical criteria, more thorough information demonstrating a product's conformance to a Criteria may result in 4 points.

The goal is to make the scoring process fair, consistent, and objective while appropriately balancing the variables involved.

Caveats

• There are product categories that are small or narrow, and vendors in these spaces may not yet address accessibility well at all. When all the options in a product space lack accessibility, the purchasing organization must determine how to provide equivalent and independent access to the purchased product's interactions. It is important to remember that purchasing organizations are generally responsible for protecting people with disabilities from discrimination when participating in a service, program, or activity.
• There are still plenty of times when a purchasing organization must perform manual testing to collect the data needed to make an informed decision. As a step between being able to vet an ACR and performing manual testing, consider an accessibility-focused demo and meeting with the vendor.
• The ACR is just one tool available for organizations to use to evaluate the accessibility of a product. A diverse set of tools will help organizations adapt to third-party marketplace realities. Accessibility-focused product demonstrations and direct testing are other methods that provide insight into candidate products' accessibility.