A presentation at GitHub Universe in in San Francisco, CA, USA by Melanie Sumner
Hi, I’m Melanie Sumner and today I’m going to speak with you about Continuous accessibility and how you can better plan for accessibility as part of your engineering practice.
I’m a decorated military veteran turned software engineer and I’m delighted to speak with you all today. Today I work as a senior design systems engineer at Hashicorp. I’m work on spec as an invited expert for the WAI-ARIA working group. I’m also a member of the Ember JS framework core team where I’ve been working on open-source accessibility solutions that scale.
Just in case you are new to accessibility, let’s run through a few common terms. When I say “accessibility”, I am referring to digital accessibility; I am specifically focused making inclusive products so people with disabilities can use what we create. You might also hear it referred to as A eleven y, which is a shortened abbreviation; there are eleven letters between the A and Y in the word accessibility. I’ll also refer to the web content accessibility guidelines, or WCAG for short, as the standard by which our applications are evaluated through the use of the Success Criteria they provide. And finally, assistive technology (AT) includes hardware or software that people with disabilities use to access the web or use mobile devices. For example, a person who is blind may use screen reader software to browse and use websites on the internet through desktop browsers or mobile devices.
I give a lot of talks about accessibility, and I usually talk about what accessibility is and things that we can do now
But this talk is different, because it is a talk about a strategic vision for the future of accessibility and automation, and how I think we can get there.
So this is my promise to you. In this talk: I will share my vision for a strategic path to continuous accessibility. I will inspire you to create something new or contribute to an effort already in progress. I will empower you to think about accessibility in a way that is more aligned with other things we already do in the business of shipping technical products.
Before we get into strategic visioning though, I want you to close your eyes for just a moment, and let yourself imagine.
I want you to imagine having greater confidence in the quality of your code.
I want you to imagine more easily delivering accessible experiences at scale.
Finally, I want you to Imagine knowing that you’ve responsibly reduced the company’s risk.
The risk of losing potential customers, The risk of decreased usage from existing customers, Or even the risk of facing accessibility lawsuits.
In software engineering today, we have continuous integration
We also have continuous deployment, and these two things have drastically improved the lives of software engineer.
But now I want us to turn our thoughts toward a new idea- continuous accessibility. How will we get there? How can we deliberately achieve this vision? We have to be on purpose.
The well-established principles of continuous software engineering remind us to build quality in
To work in small batches
To let computers do the repetitive tasks so humans can solve the hard problems
To always be improving
And that everyone is responsible for doing these things.
So how do these already-existing principles apply to accessibility? I propose that there are three key points for achieving continuous accessibility.
First, we must have a plan for the code we already have. How will we improve its accessibility?
Second, we must plan for the code we will have in the future, the code we will create.
And finally, we must have a plan for how to measure our work and show our progress.
So let’s dive into these three strategic implementation points.
Let’s think about the code we already have.
Any plans we make to improve accessibility in our products should include plans for the code we already have. We need to think about the age of our code base. How long has that code been around? This will probably make us think about those dependencies that we need to upgrade. How do we plan for upgrades? It’s great if we improve the accessibility of a common dependency, but what does this mean for existing users? What does the upgrade path for them look like? This will probably make us think about the ease…or not…of delivering the latest and greatest. We need to carefully consider how we will deliver new features or new developer tools. Backwards compatibility and stability are important and, let’s face it, sometimes overlooked in our race to build new awesome things. Unless we’ve done this thoughtfully and purposefully, the very people we are trying to empower may reject what we have to offer them.
This is especially true for apps at scale. Or even, apps that WANT to scale. It can be tricky to update a dependency especially when it includes breaking changes. Depending on the size of your codebase, taking in a new version of a dependency can mean extra developer coordination about which new features to use, or when to do the implementation work. There are also product and business priorities to consider. Features we create for accessibility automation should make it simpler to deliver improved products and tooling to support accessibility as a facet of our craft. We can’t just deliver accessibility improvements in a vacuum and expect things to go well.
For the code we have now, we could wait for our users to report issues to us- but we run the the risk of losing customers, or reduced current customer engagement.
We could also rely on audits to tell us where the issues are…but again, this is a little late in the app creation process.
Ideally, we are automating checks with linting and testing, periodically checking the code we already have and making sure that the new code we are crafting is conformant. This gives us higher level of confidence that as technology progresses, our applications still work as expected. Of course, there is still a lot of opportunity for innovation in this area.
But, this approach allows automation to be our first line of defense backed up by the accessibility audits that we might need for certain customers who require accessibility as condition of purchase and finally backed up by user reports (because users still will tell us when they run into blockers
Let’s talk about our future code. We need to plan for the code we will write in the future, Or, we will be much more likely to make the same mistakes of our past.
Who knows what will come next? The way we wrote code ten years ago is different from the way we write code today. And, who even knows how we’ll write code in ten years from now.
So what does our strategy need to prepare us for these unknown unknowns?
Of course, We will continue to follow the principles of continuous software engineering. I want to zero in on number three: letting computers perform repetitive tasks, so people can solve hard problems. In the area of accessibility automation especially, this is where we are seeing more innovation…and I think where, if I can successfully convince you, we will continue to see even more innovation in the future.
To do that, Let’s take a look at the way automation in accessibility can help.
Developers have access to automated testing for dynamic code analysis Through through the axe-core library that has been…and can be… integrated into the continuous integration and delivery mechanisms that are already used to test and deliver our code. These days, developers have a lot of options too! Axe-core has been implemented in many tools, like Lighthouse or Microsoft’s Accessibility Insights. Even EmberJS, the open-source javascript framework that I work on, has a library called ember-a11y-testing that can be integrated into an Ember app for automated accessibility testing.
For static analysis, developers can use ember-template-lint, jsx-a11y, and lit-a11y…even vue-a11y is coming along. Each have varied levels of support.
I want to talk a little bit about that. They all mostly work the same, but I’ll talk through it by talking about ember-template-lint because that the one I’m most familiar with and have been working on for a few years now. Here’s some of the benefits: Developers can get linting feedback while they are writing their code It also supports a plugin system that allows teams to define and use custom rules Some rules have automatic fixes built right in, and running the ”fix” flag will cleanup all of the auto-fixable issues Finally, it supports sharing configs across projects, so you can ensure that all of your teams are on the same page, which I think is especially really valuable in today’s globally-distributed workforce
So quick story: Before version 3.0 of ember-template-lint, the print-pending flag gave us a way to roll out new rules. you could take a list of current errors and basically make an “ignore these errors” list in the linter- but it relied entirely on teams to be proactive and treat that list as a burndown list, not an “ignore permanently forever” list. Of course, quality engineering teams understood that we didn’t really want them to ignore these errors forever, but it’s easy to forget about something that doesn’t have a deadline.
One approach was to turn a rule off completely until all existing instances of that error were fixed. Of course, this led to a never ending cycle of trying to turn a rule on, finding new bugs, fixing those bugs…and starting all over again This is because by the time you fixed all the errors, new code would appear with those same errors. Not only is this approach time-consuming and costly… it can really disincentivize teams to keep up with best practices.
So we had issues that could go into the lost void of forever Or stuck in a cycle of always almost ready to turn a rule on We needed to do something different.
Enter support for the TODO. Instead of only having the option to set a rule to warning or error, We can now instruct the linter to find all existing instances of a rule being broken, and create a “todo” for each. This improves on the previous approach in a couple of ways. After a period of time, this “todo” turns into a warning And then an error (which acts as a forcing function- the build breaks.
If we look at this from the perspective of the code we already have and the code we will create, we see that we can break the problem into different parts, so we can immediately benefit.
When a new lint rule is released,
Now you might be wondering, okay but can’t teams just create a new todo? And the answer is, yes, of course they can.
This is about empowering developer teams to move faster. This is about giving PMs the right information so they can make the best planning decisions to support their teams. This is about guidance, not control.
On to the third key part of our strategy- plan to measure our progress. Right now, there is no common, shared standard for metrics in accessibility engineering; my vision is to change that. Metrics play an essential part of our strategy. After all, metrics and reporting is how we empower our senior management (and other decision makers) to give us what we need. Things like…time. Things like budget. Things like vocalization of priorities. When we provide ways to measure this work, we are bridging a very important gap.
Just in case you haven’t had to work with metrics yet, or haven’t in a while, let’s review four of the commonly accepted key criteria of quality metrics: Meaningful. They must be connected to the goals and/or strategy of our organization. Controllable. If a metric is not under your control or influence, it is not meaningful to report (even if it is useful to know about). Easy to Access. Metrics should be easy to find, easy to identify and visible to management Actionable. Any metrics that we define MUST have actionable outcomes.
but what do we mean when we say actionable outcomes? well, we can diagnose a problem improve a process set a goal or observe trends that can inform future work
But before I get into the specific metrics that I think we should be measuring, I want to remind us of Goodhart’s Law- our metrics should never themselves become the entire point. We won’t achieve the outcomes we are after (improved accessibility and user experience) if we merely game the system.
There’s a balance to be had; we need to measure things in a business environment, but we need to deliver quality for our users, not only meet a metric or fulfill a WCAG Success Criterion.
First metric: Potential Violation Count. This metric sets the baseline for us. Now, we all know that there are an infinite number of ways that things can go wrong in any given application. But that isn’t what we mean here. With potential violation count, we want to make the unknown problem a known problem, then solve for each. The baseline is the total number of individual ways an application could fail the legal accessibility requirements. but how do we get that itemized number? In WCAG we have success criteria, known techniques, and common failures. There are also location-specific Legal Standards like Section 508 or the Americans with Disabilities Act. Of course, all of these potential violations represent massive effort that has already been completed by user researchers but how do you get that information in a practical way?
We do this by making an itemized list. There will, of course, be some overlap, but this work is worth doing because it gives us the peace of mind that we really know the edges of this problem. We’re turning an area of ambiguity into an area of clarity We’re making the unknown edges of the problem known and that can help give us confidence.
Before you think “oh my gosh that’s a lot of work” – let me share an effort that has already begun to itemize potential violations. It’s an open source project called the a11y automation tracker, and intends to compile each one of these itemized potential violations with details.
There is an overall detail list, but there is also the ability to dive into each potential violation and see the details about how it can currently be tested,
The existing linting rules and the libraries they are in The testing rules and the libraries they are in, and the relevant WCAG Success Criteria, with links do documentation. Of these potential failures, my initial analysis has indicated that about half of them are either already automated or are potentially automatable.
but what about the other half? This means that the rest still require manual testing, until technology sufficiently advances to programmatically determine these errors. For example: how do we codify intent? In a lot of cases, we still have the opportunity to innovate there. But by taking advantage of the automation that IS available to us, we can reduce the need for manual testing by nearly half, and that’s A LOT less manual testing to be done.
Some of you who are already familiar with the WCAG Success Criteria might be wondering, why not just use those? Well, it’s because they cover generalities rather than specifics. For example, WCAG 1.3.1 (Info and Relationships) is a single success criterion but relates to at least 25+ different failure scenarios. Linting rules have to cover one specific failure, catching as many potential implementation syntax errors as possible
By identifying the edges of the potential violation count, we can then determine related metrics:
Violations for which we can provide automated linting for static analysis Violations for which we can provide automated testing for dynamic analysis Violations that require developer-authored tests Violations that require manual testing These metrics are useful if we build tooling! Once we set a baseline we can target improvements.
There are also metrics that we can look at from the accessibility audits we receive.
What is the Total Bug Count? Bug Severity Count – how severe of an impact do each of these bugs have for our users? For example, a nested interactive element, like a link inside of a button, will result in a complete blocker for a user with assistive tech. On the other hand, a main element with the role of main does not harm the user, it’s merely redundant code. Time to fix- how long does it take a team to resolve an issue? This can be a signifier of a process issue within a team or org, or perhaps a signifier that accessibility hasn’t been vocalized as a priority. If we track this metric, we can start to look at potential causes to help make our apps healthier…in a way we’re already familiar with. And finally, let’s measure violation frequency. In most modern apps, we use components, so maybe we can track what components need to be updated. Or maybe we’ll see that a specific team is contributing a violation more frequently than others. There’s an opportunity to level up our engineers!
So, I’ve given us some essential metrics to track but what sorts of things are we hoping to learn? What impact do we expect to have? We should expect, at a minimum, to see trends as a result of the actions we take.
If we work on tooling, we could expect to see an increase in the number of automated linting rules and tests, since we have a better way to identify specific possible violations. We can also expect to see developers writing more accessibility related tests, since we’re providing them more guidance and education.
we can also expect to see a decrease in support requests for accessibility experts within the org, because developers will have more linting and testing coverage, and a greater awareness of accessibility in general. This awareness should lead to fewer a11y-related issues in new code (specifically, issues we can’t write an automated test for yet) And since we’ve turned automation on, we should see fewer issues in code that has the new automation applied to it
Our audit-related metrics could also inform process from a business/legal perspective, as they can be used to help quantify risk of legal action. When the risk is quantified, we can then quantify the reduction in that risk by taking specific steps to remedy the issues; while this may be a business justification, the result is that our users have an improved member experience, which seems like a win-win. Improved monitoring means that development teams can know sooner if a product’s conformance deteriorates at a rate outside of acceptable thresholds. (Conformance Deterioration Rate (CDR)). It can help us determine a threshold for new learning opportunities. If a high percentage of developers keep writing non-conformant code in a specific way, we can produce materials that will help them on their learning journey We can also use these trends to determine an app’s overall accessibility health. Maybe even put that in a….oh what’s that thing people really love to use at work? Reporting dashboards?
So…a reporting dashboard. Let’s imagine that.
Imagine being able to check in on a product to see how it’s doing and get all of that information easily and quickly.
It could be used to inform decisions: rationalize time, budget, etc. It could be used to demonstrate that an investment in accessibility tooling for designers and engineers could make a big difference. It could be used by sales teams when a potential client wants to know about our app’s accessibility conformance Frankly, there are a lot of use cases but they all come down to this: everyone will be more informed about the decisions they make.
In addition to trend analysis, we can use the metrics we gather to inform future work. Like, Which potential violations currently require manual testing but could reasonably be automated? Just hasn’t been done yet? What violations happen the most? Is there something we could create to make that problem go away or make it easier? How could we make it faster for developers to fix the issues that seem to take a long time to resolve? Is it a code problem? Is it a tooling challenge? Is it a problematic process?
I think there could reasonably be additional metrics to be considered here, but the ones we’ve discussed give us a solid start to quantifying accessibility in our engineering practice, bringing us closer to continuous accessibility.
I think the single biggest thing that we can give not only ourselves but also our senior management is information. The information they need to support us from a business perspective.
The thing is, accessibility IS a civil right, and it is also an emotional issue. And that seems like it’s at odds with business IN GENERAL. But I envision an environment where it doesn’t have to be.
We can accept, as a company value, that accessibility is a facet of quality. We can also rationalize the work that needs doing…with numbers. Just like every other part of the business.
Maybe in a perfect world, we wouldn’t have to do these things, but also…we CAN do this, so why not give business what it needs? After all, the outcomes are what matters.
You can use sites like Please fund a-eleven-y dot com a website that provides a list of the known areas where funding or support would improve the state of accessibility across the web. Maybe your company has engineer time that can be used to help solve some of these issues. Maybe your company has funding that can be allocated to hire technical writers to improve the state of accessibility documentation.
Or, perhaps you can contribute to some of the open-source projects that exist to provide linting rules for the framework your company uses.
Another potential outcome is to develop training materials that help relate the information that people already know to accessibility concepts that they maybe don’t, like this training project that I’ve been working on at HashiCorp to help relate existing design heuristics with accessibility concepts
There are so many possible outcomes here…it’s like a choose your own adventure! There is room for everyone to contribute.
At the beginning of this talk, I made a few promises. So let’s see if I kept them. As we have looked at how to think about our code and what metrics to consider, I hope you have seen the path forward for continuous accessibility. I hope I have inspired you to create something new or contribute to an effort already in progress. I hope I have empowered you to think about accessibility in your design or code or even in your process in a new way.
And if you’re a decision maker, I hope I have shown you that continuous accessibility is the thing you’ve been waiting for. Think SRE, but A11y.
Continuous accessibility is good for business, but it’s great for users. And that’s what we’re really going for.
Thank you!!
What needs to happen for continuous accessibility to be just as much of a standard practice as continuous integration? In this session, Melanie Sumner will look at a strategic vision for the future of the web that includes a more inclusive web for everyone.
