A presentation at a11yTO in October 2022 in Toronto, ON, Canada by Melanie Sumner
Hello! I’m Melanie Sumner, a decorated, disabled military veteran turned software engineer and I’m delighted to speak with you all today about the idea of continuous accessibility and how we can better plan for accessibility in our engineering practice.
If you’d prefer to view this deck on your personal device, it’s available at https noti dot st forward slash melsumner forward slash b n g y m o.
If we haven’t met yet, here’s a few things about me- I’m an invited expert for the WAI-ARIA working group,
I lead accessibility initiatives for the Ember JavaScript Framework as a member of the core team,
And I’m a board member for Vets Who Code, an organization that retrains military veterans so they can land jobs in tech as civilians.
But my passion is digital accessibility solutions that scale. And stardew valley.
For my day job, I am senior software engineer on the design system team at HashiCorp, where we are focused on empowering designers and developers at HashiCorp to produce accessible products.
HashiCorp is a cloud devops company, and the products we create are used by many companies across the world
my personal motivation is that I don’t want anyone to miss out on a job at a company like GitHub, Bloomberg, Eventbrite or HashiCorp itself because the HashiCorp platform they’d need to use at work is not accessible.
So I usually talk to a room of JavaScript Engineers, and have learned over the years to have a TL;DR slide so if they are entirely new to this concept they don’t feel totally lost
But since this audience maybe probably doesn’t need a primer, I’ll skip that slide and instead talk about..who is this talk for? This talk is for everyone in this room, but this joke is for Simpson’s fans. This talk is for especially engineers but especially managers And of course, anyone else who has to work with them and convince them to do things.
I give a lot of talks about accessibility, most of which are available on my notist profile and I usually talk about what accessibility is and things that we can do now
But this talk is different, because it is a talk about a strategic vision for the future of accessibility and automation, and how I think we can get there.
My talk framework tells me that I’m supposed to let you know what you can expect to learn from this talk in the form of promises. Just regular promises, no JavaScript.
So these are my promises to you. In this talk, I will share my vision for a strategic path to continuous accessibility.
I will inspire you to create something new or contribute to an effort already in progress.
I will empower you to think about accessibility in a way that is more aligned with other things we already do in the business of shipping technical products.
Before we get into strategic visioning though, I want you to take a moment, and let yourself imagine.
I want you to imagine having greater confidence in the quality of your code.
I want you to imagine more easily delivering accessible experiences at scale
Finally, I want you to Imagine knowing that you’ve reduced risk.
The risk of losing potential customers, The risk of decreased usage from existing customers, Or even the risk of facing accessibility lawsuits.
In software engineering today, we have continuous integration
we also have continuous deployment And these two things have drastically improved the lives of software engineers.
But now I want us to turn our thoughts toward a new idea- continuous accessibility. How will we get there? How can we achieve this vision?
We have to be on purpose.
The well-established principles of continuous software engineering have taught us to build quality in.
To work in small batches.
Let computers solve repetitive tasks so people can solve hard problems.
to always be improving
and that everyone is responsible.
So. I propose that there are three key points for achieving continuous accessibility.
First, we must have a plan for the code we already have. Will we remediate, deprecate, or burn it all down?
Second, we must plan for the code we will have in the future, the code we will create.
And third, we must have a plan for how to measure our work and show our progress.
So let’s dive into these three facets of our implementation strategy.
Let’s think about the code we already have.
Any plans we make to improve accessibility in our products should include plans for the code we already have. We need to think about the age of our code base. How long has that code been around?
This will probably make us think about those dependencies that we need to upgrade. How do we plan for upgrades? It’s great if we improve the accessibility of a common dependency, but what does this mean for existing users? What does the upgrade path for them look like?
This will probably make us think about the ease…or not…of delivering the latest and greatest. We need to carefully consider how we will deliver new features or new developer tools. Backwards compatibility and stability are important and, let’s face it, sometimes overlooked in our race to build new awesome things. Unless we’ve done this thoughtfully and purposefully, the very people we are trying to empower may reject what we have to offer them.
This is especially true for apps at scale. Or even, apps that WANT to scale. It can be tricky to update a dependency especially when it includes breaking changes. Depending on the size of your codebase, taking in a new version of a dependency can mean extra developer coordination about which new features to use, or when to do the implementation work. There are also product and business priorities to consider.
Features we create for accessibility automation should make it simpler to deliver improved products and tooling to support accessibility as a facet of our craft. We can’t just deliver accessibility improvements in a vacuum and expect things to go well.
For the code we have now, we could wait for our users to report issues to us- but we run the the risk of losing customers, or reduced current customer engagement.
We could also rely on audits to tell us where the issues are…but again, this is a little late in the app creation process.
Ideally, we are automating checks with linting and testing, periodically checking the code we already have and making sure that the new code we are crafting is conformant. This gives us higher level of confidence that as technology progresses, our applications still work as expected. Of course, there is still a lot of opportunity for innovation in this area.
But, this approach makes automation our first line of defense backed up by the accessibility audits that we might need for certain customers who perhaps require accessibility as condition of purchase and finally backed up by user reports (because users still will tell us when they run into blockers
Let’s talk about our future code. We need to plan for the code we will write in the future, Or, we will be much more likely to make the same mistakes of our past.
Who knows what will come next? The way we wrote code ten years ago is different from the way we write code today. And, who even knows how we’ll write code in ten years from now.
So what does our strategy need to prepare us for these unknown unknowns?
Of course, We will continue to follow the principles of continuous software engineering. I want to zero in on number three: letting computers perform repetitive tasks, so people can solve hard problems. In the area of accessibility automation especially, this is where we are seeing more innovation…and I think where, if I can successfully convince you, we will continue to see even more innovation in the future.
To do that, Let’s take a look at the way automation in accessibility can help.
There are testing libraries for dynamic code analysis. Through through the axe-core library (pause) that has been integrated into many continuous integration and delivery mechanisms that are already used to test and deliver our code. These days, developers have a lot of options too! Axe-core has been implemented in many tools, like Lighthouse or Microsoft’s Accessibility Insights. Even EmberJS, the open-source javascript framework that I work on, has a library called ember-a11y-testing that can be integrated into an Ember app for automated accessibility testing.
To complement that, there are linters for many different javascript frameworks that give developers real-time feedback while they are writing their code. Each has varied levels of support; Ember comes in with the most a11y-related linting rules, while vue has the fewest.
Now They all mostly work the same, but I’ll talk through it by talking about ember-template-lint because that the one I’m most familiar with and have been working on for a few years now. Here’s some of the benefits: Developers can get feedback while they are writing their code It also supports a plugin system that allows teams to define and use custom rules Some rules have automatic fixes built right in, and running the ”fix” flag will cleanup all of the auto-fixable issues Finally, it supports sharing configs across projects, so you can ensure that all of your teams are on the same page, which I think is especially really valuable in today’s globally-distributed workforce I especially like pre-decided things because there’s always this annoying engineer and Steve always tries to insist that everyone else does things his way, so if there’s a config that’s one less argument you have to have with him.
So quick story: Before version 3.0 of ember-template-lint, the print-pending flag gave us a way to roll out new rules. you could take a list of current errors and basically make an “ignore these errors” list in the linter- but it relied entirely on engineering teams to be proactive and treat that list as a burndown list, not an “ignore permanently forever” list. Of course, quality engineering teams understood that we didn’t really want them to ignore these errors forever, but it’s easy to forget about something that doesn’t have a deadline.
One approach was to turn a rule off completely until all existing instances of that error were fixed. Of course, this led to a never ending cycle of trying to turn a rule on, finding new bugs, fixing those bugs…and starting all over again This is because by the time you fixed all the errors, new code would appear with those same errors. Not only is this approach time-consuming and costly… it can really disincentivize teams to keep up with best practices.
So we had issues that could go into the lost void of forever Or stuck in a cycle of always almost ready to turn a rule on We needed to do something different.
Enter support for the TODO. Instead of only having the option to set a rule to warning or error, We can now instruct the linter to find all existing instances of a rule being broken, and create a “todo” for each. This improves on the previous approach in a couple of ways. After a period of time, this “todo” turns into a warning And then an error (which acts as a forcing function- the build breaks.
If we look at this from the perspective of the code we already have and the code we will create, we see that we can break the problem into different parts, so we can immediately benefit. When a new lint rule is released,
Now you might be wondering, okay but can’t teams just create a new todo? And the answer is, yes, of course they can. Because this is about empowering developer teams to move faster. This is about giving managers the right information so they can make the best planning decisions to support their teams. This is about guidance, not control.
On to the third key part of our strategy- plan to measure our progress. Right now, there is no commonly shared standard for metrics in accessibility engineering; my vision is to change that. Metrics play an essential part of our strategy. After all, metrics and reporting is how we empower our senior management (and other decision makers) to give us what we need. Things like…time. Things like budget. Things like vocalization of priorities. When we provide ways to measure this work, we are bridging a very important gap.
Just in case you haven’t had to work with metrics yet, or haven’t in a while, let’s review four of the commonly accepted key criteria of quality metrics: Meaningful. They must be connected to the goals and/or strategy of our organization. Controllable. If a metric is not under your control or influence, it is not meaningful to report (even if it is useful to know about). Easy to Access. Metrics should be easy to find, easy to identify and visible to management Actionable. Any metrics that we define MUST have actionable outcomes.
but what do we mean when we say actionable outcomes? well, we can diagnose a problem improve a process set a goal or observe trends that can inform future work
But before I get into the specific metrics that I think we should be measuring, I want to remind us of Goodhart’s Law- When a measure becomes a target, it ceases to be a good measure. our metrics should never themselves become the entire point. We won’t achieve the outcomes we are after (improved accessibility and user experience) if we merely game the system.
There’s a balance to be had; we need to measure things in a business environment, but we need to deliver quality for our users, not only meet a metric or fulfill a WCAG Success Criterion.
First metric: Potential Violation Count. This metric sets the baseline for us. Now, we all know that there are an infinite number of ways that things can go wrong in any given application. But that isn’t what we mean here. With potential violation count, we want make the unknown problem a known problem, then solve for each. The baseline is the total number of individual ways an application could fail the legal accessibility requirements. but how do we get that itemized number? In WCAG we have success criteria, known techniques, and common failures. There are also location-specific Legal Standards like Section 508 or the Americans with Disabilities Act. Of coursem all of these potential violations represent massive effort that has already been completed by user researchers but how do you get that information in a practical way?
We do this by making an itemized list. There will, of course, be some overlap, but this work is worth doing because it gives us the peace of mind that we really know the edges of this problem. We’re turning an area of ambiguity into an area of clarity We’re making the unknown edges of the problem known and that can help give us confidence.
Before you think “oh my gosh that’s a lot of work” – let me share an effort that has already begun to itemize potential violations. It’s an open source project called the a11y automation tracker, and intends to compile each one of these itemized potential violations with details.
There is an overall detail list, but there is also the ability to dive into each potential violation and see the details about how it can currently be tested,
The existing linting rules and the libraries they are in The testing rules and the libraries they are in, and the relevant WCAG Success Criteria, with links do documentation. Of these potential failures, my initial analysis has indicated that about half of them are either already automated or are potentially automatable.
but what about the other half? This means that the rest still require manual testing, until technology sufficiently advances to programmatically determine these errors. For example: how do we codify intent? In a lot of cases, we still have the opportunity to innovate there. But by taking advantage of the automation that IS available to us, we can reduce the need for manual testing by nearly half, and that’s A LOT less manual testing to be done.
Some of you who are already familiar with the WCAG Success Criteria might be wondering, why not just use those? Well, it’s because they cover generalities rather than specifics. For example, WCAG 1.3.1 (Info and Relationships) is a single success criterion but relates to at least 25+ different failure scenarios. Linting rules have to cover one specific failure, catching as many potential implementation syntax errors as possible
By identifying the edges of the potential violation count, we can then determine related metrics: Violations for which we can provide automated linting for static analysis Violations for which we can provide automated testing for dynamic analysis Violations that require developer-authored tests Violations that require manual testing These metrics are useful if we build tooling! Once we set a baseline we can target improvements. These metrics could also help a company decide which framework to use for their code. Which one sets them up to be successful in building accessible products?
There are also metrics that we can look at from the accessibility audits we receive.
What is the Total Bug Count? Bug Severity Count – how severe of an impact do each of these bugs have for our users? For example, a nested interactive element, like a link inside of a button, will result in a complete blocker for a user with assistive tech. On the other hand, a main element with the role of main does not harm the user, it’s merely redundant code. Time to fix- how long does it take a team to resolve an issue? This can be a signifier of a process issue within a team or org, or perhaps a signifier that accessibility hasn’t been vocalized as a priority. If we track this metric, we can start to look at potential causes to help make our apps healthier…in a way we’re already familiar with. And finally, let’s measure violation frequency. In most modern apps, we use components, so maybe we can track what components need to be updated. Or maybe we’ll see that a specific team is contributing a violation more frequently than others. There’s an opportunity to level up our engineers!
So, I’ve given us some essential metrics to track but what sorts of things are we hoping to learn? What impact do we expect to have? We should expect, at a minimum, to see trends as a result of the actions we take. If we work on tooling, we could expect to see an increase in the number of automated linting rules and tests, since we have a better way to identify specific possible violations. We can also expect to see developers writing more accessibility related tests, since we’re providing them more guidance and education.
we can also expect to see a decrease in support requests for accessibility experts within the org, because developers will have more linting and testing coverage, and a greater awareness of accessibility in general. This awareness should lead to fewer a11y-related issues in new code (specifically, issues we can’t write an automated test for yet) And since we’ve turned automation on, we should see fewer issues in code that has the new automation applied to it
Our audit-related metrics could also inform process from a business/legal perspective, as they can be used to help quantify risk of legal action. When the risk is quantified, we can then quantify the reduction in that risk by taking specific steps to remedy the issues; while this may be a business justification, the result is that our users have an improved member experience, which seems like a win-win. Improved monitoring means that development teams can know sooner if a product’s conformance deteriorates at a rate outside of acceptable thresholds. (Conformance Deterioration Rate (CDR)). It can help us determine a threshold for new learning opportunities. If a high percentage of developers keep writing non-conformant code in a specific way, we can produce materials that will help them on their learning journey We can also use these trends to determine an app’s overall accessibility health. Maybe even put that in a….oh what’s that thing people really love to use at work? Reporting dashboards?
So…a reporting dashboard. Let’s imagine that.
Imagine being able to check in on a product to see how it’s doing and get all of that information easily and quickly.
It could be used to inform decisions: rationalize time, budget, etc. It could be used to demonstrate that an investment in accessibility tooling for designers and engineers could make a big difference. It could be used by sales teams when a potential client wants to know about our app’s accessibility conformance Frankly, there are a lot of use cases but they all come down to this: everyone will be more informed about the decisions they make.
In addition to trend analysis, we can use the metrics we gather to inform future work. Like, Which potential violations currently require manual testing but could reasonably be automated? Just hasn’t been done yet? What violations happen the most? Is there something we could create to make that problem go away or make it easier? How could we make it faster for developers to fix the issues that seem to take a long time to resolve? Is it a code problem? Is it a tooling challenge? Is it a problematic process?
I think there could reasonably be additional metrics to be considered here, but the ones we’ve discussed give us a solid start to quantifying accessibility in our engineering practice, bringing us closer to continuous accessibility.
I think the single biggest thing that we can give not only ourselves but also our senior management is information. The information they need to support us from a business perspective.
The thing is, accessibility IS a civil right, and it is also an emotional issue. And that seems like it’s at odds with business IN GENERAL. But I envision an environment where it doesn’t have to be.
We can accept, as a company value, that accessibility is a facet of quality. We can also rationalize the work that needs doing…with numbers. Just like every other part of the business.
Maybe in a perfect world, we wouldn’t have to do these things, but also…we CAN do this, so why not give business what it needs? After all, the outcomes are what matters.
In focusing on the outcomes, you can also come up with other ideas to move accessibility forward. You can use sites like Please fund a-eleven-y dot com a website that provides a list of the known areas where funding or support would improve the state of accessibility across the web. Maybe your company has engineer time that can be used to help solve some of these issues. Maybe your company has funding that can be allocated to hire technical writers to improve the state of accessibility documentation.
Or, if you write code, perhaps you can contribute to some of the open-source projects that exist to provide linting rules for the framework your project uses.
Another potential outcome is to develop training materials that help relate the information that people already know to accessibility concepts that they maybe don’t, like this training project that I’ve been working on at HashiCorp to help relate existing design heuristics with accessibility concepts
There are so many possible outcomes and potential action steps here…it’s like a choose your own adventure! There is room for everyone to contribute.
At the beginning of this talk, I made a few promises. So let’s see if I kept them. As we have looked at how to think about our code and what metrics to consider, I hope you have seen the path forward for continuous accessibility. I hope I have inspired you to create something new or contribute to an effort already in progress. I hope I have empowered you to think about accessibility in your design or code or even in your process in a new way.
And if you’re a decision maker, I hope I have shown you that continuous accessibility is the thing you’ve been waiting for. Think SRE, but A11y.
Continuous accessibility is good for business, but it’s great for users. And that’s what we’re really going for.
Thank you for your time today! Please tweet at me and let me know what you thought about this talk, I’m at A11y Mel on Twitter!
