CSUN 2023 Analytics.pptx

Editor's Notes

1. Elsevier helps researchers and healthcare professionals advance science and improve health outcomes for the benefit of society. We are the largest publisher of books and journals in Science Prolific publisher in disability research and accessibility Lancet, Cell, ScienceDirect, Scopus, Clinical Key Books by Jonathan Lazar
2. Ted: could play managing director of research products business unit. Hearing great things about the accessibility team. Thank you. As our leadership team is planning our objectives and key results for this year, could you help identify some accessibility metrics. Let’s start by establishing a baseline accessibility metric and then set an achievable target for teams to meet. Try to come up with something in lay person’s terms. See you in a month. So that was a skit based upon a recurring question our a11y team receives, typically from PMs to help establish meaningful metrics around accessibility. So how do we identify some accessibility metrics, how go about that?
3. Before we help the managing director and need for simple metrics, first let’s define the nature of accessibility conformance data. And to do that let’s also go through some terms. Has anyone ever heard of a VPAT? Voluntary Product Accessibility Template, ACR = Accessibility Conformance Report? An accessibility expert uses the VPAT to evaluate the accessibility conformance of a digital product. This is a blank VPAT, it shows a cover page of the WCAG edition and the success criteria overview for WCAG 2.1.
4. Taking an example checkpoint from the VPAT, here’s 1.1.1 with a grading of partially supports. The Remarks provides a narrative of supporting remarks and exceptions. Some icons are labeled with alt text, but there are some exceptions.
5. When completed, the ACR represents 50 checkpoints and serves as like the nutrition facts of accessibility for a digital product. Ours are about 12 pages in length. This particular ACR shows a website that is mostly compliant. Of the 50 WCAG 38 fully supported, 10 partially supports, and 2 does not support. Now with all of this detail, is it easy to explain if 1 website has improved since last VPAT? How about a program tracking 50 or 100 websites? Keep in mind there isn’t an official scoring system provided with VPATs
6. So what is the problem then we are trying to solve? The problem is the mismatch between the nature of mostly qualitiatve ACR accessibility data and the need for simple metrics. As in our skit, managers want to know simple easy to use scores and answers questions like “did my products improve since last VPAT evaluation”? Important decision making. What if a website owner has a false sense of security that they have good accessibility? We want to inform decisions makers with analytics to help improve and maintain high levels of accessibility How then? With some metrics.
7. So what did we do to help solve the problem of needing to create simple quantifiable metrics around accessibility? Created our own excel database using the gradings for each WCAG checkpoints excerpted ACRs. Back in 2019 we started with 38 rows of 38 WCAG 2.0, 1444 data points. As shown in this example table, each product has a row in the Excel sheet reflecting the evaluation from the ACR. Each product shows the total number of checkpoints (fully supports, partially supports, does not support) We provide a count and a percentage of the total number of WCAG checkpoints. This db is the basis of our analytics work
8. For each product we also added the individual ratings of each of the 50 WCAG criteria. This then allowed us to total the numbers of fully supported, partially supported, and not supported checkpoints across products. So for this table it shows that for non text content, 6 products fully support non-text content, 57 partially support, and 5 do not support
9. What did creating the database allow us to do? There was some warming up to Excel. This data provided a new capability to compare VPAT ratings between products and rank them. And ranking became a new part of our topline VPAT service for the company. Also allowed teams the ability to set a benchmark, let’s say to improve a score from 70% to 80%. Guess what we have been asked by customers to meet a certain fully supported score Jay next slides
10. TO JAY – I added this slide from Astrid to illustrate how our work translates into a real annual objective. If you like it we can keep it PM quote related to the overall analytics…  “The ability to see how our site compares to other sites in the company is great motivation for the squad, they pride themselves on having a #1 position”   PM quote Related to Evolve analytics…  “Seeing where our site excels in accessibility and where it falls short makes it easy for us to prioritize and plan improvements”
11. TG In Summary: Our problem statement is now more of a solution statement instead of an aim We inform decisions makers with analytics to help improve and maintain high levels of accessibility Our approach created a new layer of intelligence around our ACRs new comparison ability that didn’t exist before Driving competition between teams is good Motivates New way to view most commons problem WCAG areas
12. Ring, Hi Jay, this is Ted Gies, Managing Director of Research Products. How’s everything Very curious to know what you’ve been working on Ted looking very interested in Jay’s report Great Objective Setting Call you again next month?
13. *MAYBE LOSE THIS SLIDE OR PUT POINTS INTO THE DRAWBACKS SLIDE* At Elsevier we use VPATs for business and measure Eventhough there are other ways to measure A11y and usasbility, user testing the truth. The VPAT is our preferred benchmark because 1) it’s what our customers ask for (US Govt, Colleges around the world). 2) it’s comprehensive and in line with international standards. We’ve been creating VPATs since 2007 to the delight of 100s of customers. We like VPATs because they provide the full compliance picture allowing us to use measure accessibility using manual and automated tools to give a well rounded view against WCAG 2.1. VPAT is one lens of 4 we use around a11y, we want to use all of these lenses, each will ensure accessibility. There are data analytics available in tools (ARC, PopeTech) however all based upon machine testable criterion (20%-30% can be automated of the story) 37 of WCAG 2.1 cannot be automated (Sensory characteristics, Pointer Gestures, Error Prevention).