GitHub Universe: Continuous Accessibility

A presentation at GitHub Universe in November 2022 in San Francisco, CA, USA by Melanie Sumner

Slide 1

Slide 1

Continuous Accessibility

Hi, I’m Melanie Sumner and today I’m going to speak with you about Continuous accessibility and how you can better plan for accessibility as part of your engineering practice.

Slide 2

Slide 2

Intro

I’m a decorated military veteran turned software engineer and I’m delighted to speak with you all today. Today I work as a senior design systems engineer at Hashicorp. I’m work on spec as an invited expert for the WAI-ARIA working group. I’m also a member of the Ember JS framework core team where I’ve been working on open-source accessibility solutions that scale.

Slide 3

Slide 3

Accessibility Primer

Just in case you are new to accessibility, let’s run through a few common terms. When I say “accessibility”, I am referring to digital accessibility; I am specifically focused making inclusive products so people with disabilities can use what we create. You might also hear it referred to as A eleven y, which is a shortened abbreviation; there are eleven letters between the A and Y in the word accessibility. I’ll also refer to the web content accessibility guidelines, or WCAG for short, as the standard by which our applications are evaluated through the use of the Success Criteria they provide. And finally, assistive technology (AT) includes hardware or software that people with disabilities use to access the web or use mobile devices. For example, a person who is blind may use screen reader software to browse and use websites on the internet through desktop browsers or mobile devices.

Slide 4

Slide 4

I talk about a11y

I give a lot of talks about accessibility, and I usually talk about what accessibility is and things that we can do now

But this talk is different, because it is a talk about a strategic vision for the future of accessibility and automation, and how I think we can get there.

Slide 5

Slide 5

My Promise

So this is my promise to you. In this talk: I will share my vision for a strategic path to continuous accessibility. I will inspire you to create something new or contribute to an effort already in progress. I will empower you to think about accessibility in a way that is more aligned with other things we already do in the business of shipping technical products.

Slide 6

Slide 6

Imagine

Before we get into strategic visioning though, I want you to close your eyes for just a moment, and let yourself imagine.

Slide 7

Slide 7

More Confidence

I want you to imagine having greater confidence in the quality of your code.

Slide 8

Slide 8

Scale

I want you to imagine more easily delivering accessible experiences at scale.

Slide 9

Slide 9

Reduce Risk

Finally, I want you to Imagine knowing that you’ve responsibly reduced the company’s risk.

The risk of losing potential customers, The risk of decreased usage from existing customers, Or even the risk of facing accessibility lawsuits.

Slide 10

Slide 10

Continuous Integration

In software engineering today, we have continuous integration

Slide 11

Slide 11

Continuous Deployment

We also have continuous deployment, and these two things have drastically improved the lives of software engineer.

Slide 12

Slide 12

Continuous...Accessibility?

But now I want us to turn our thoughts toward a new idea- continuous accessibility. How will we get there? How can we deliberately achieve this vision? We have to be on purpose.

Slide 13

Slide 13

Build Quality In

The well-established principles of continuous software engineering remind us to build quality in

Slide 14

Slide 14

Small Batches

To work in small batches

Slide 15

Slide 15

Computers v. Humans

To let computers do the repetitive tasks so humans can solve the hard problems

Slide 16

Slide 16

Improve Always

To always be improving

Slide 17

Slide 17

Everyone participates

And that everyone is responsible for doing these things.

Slide 18

Slide 18

Continuous Accessibility Strategy

So how do these already-existing principles apply to accessibility? I propose that there are three key points for achieving continuous accessibility.

Slide 19

Slide 19

First, we must have a plan for the code we already have. How will we improve its accessibility?

Slide 20

Slide 20

Second, we must plan for the code we will have in the future, the code we will create.

Slide 21

Slide 21

And finally, we must have a plan for how to measure our work and show our progress.

Slide 22

Slide 22

So let’s dive into these three strategic implementation points.

Let’s think about the code we already have.

Slide 23

Slide 23

Any plans we make to improve accessibility in our products should include plans for the code we already have. We need to think about the age of our code base. How long has that code been around? This will probably make us think about those dependencies that we need to upgrade. How do we plan for upgrades? It’s great if we improve the accessibility of a common dependency, but what does this mean for existing users? What does the upgrade path for them look like? This will probably make us think about the ease…or not…of delivering the latest and greatest. We need to carefully consider how we will deliver new features or new developer tools. Backwards compatibility and stability are important and, let’s face it, sometimes overlooked in our race to build new awesome things. Unless we’ve done this thoughtfully and purposefully, the very people we are trying to empower may reject what we have to offer them.

Slide 24

Slide 24

This is especially true for apps at scale. Or even, apps that WANT to scale. It can be tricky to update a dependency especially when it includes breaking changes. Depending on the size of your codebase, taking in a new version of a dependency can mean extra developer coordination about which new features to use, or when to do the implementation work. There are also product and business priorities to consider. Features we create for accessibility automation should make it simpler to deliver improved products and tooling to support accessibility as a facet of our craft. We can’t just deliver accessibility improvements in a vacuum and expect things to go well.

Slide 25

Slide 25

For the code we have now, we could wait for our users to report issues to us- but we run the the risk of losing customers, or reduced current customer engagement.

Slide 26

Slide 26

We could also rely on audits to tell us where the issues are…but again, this is a little late in the app creation process.

Slide 27

Slide 27

Ideally, we are automating checks with linting and testing, periodically checking the code we already have and making sure that the new code we are crafting is conformant. This gives us higher level of confidence that as technology progresses, our applications still work as expected. Of course, there is still a lot of opportunity for innovation in this area.

Slide 28

Slide 28

But, this approach allows automation to be our first line of defense backed up by the accessibility audits that we might need for certain customers who require accessibility as condition of purchase and finally backed up by user reports (because users still will tell us when they run into blockers

Slide 29

Slide 29

Let’s talk about our future code. We need to plan for the code we will write in the future, Or, we will be much more likely to make the same mistakes of our past.

Who knows what will come next? The way we wrote code ten years ago is different from the way we write code today. And, who even knows how we’ll write code in ten years from now.

So what does our strategy need to prepare us for these unknown unknowns?

Slide 30

Slide 30

Of course, We will continue to follow the principles of continuous software engineering. I want to zero in on number three: letting computers perform repetitive tasks, so people can solve hard problems. In the area of accessibility automation especially, this is where we are seeing more innovation…and I think where, if I can successfully convince you, we will continue to see even more innovation in the future.

To do that, Let’s take a look at the way automation in accessibility can help.

Slide 31

Slide 31

Developers have access to automated testing for dynamic code analysis Through through the axe-core library that has been…and can be… integrated into the continuous integration and delivery mechanisms that are already used to test and deliver our code. These days, developers have a lot of options too! Axe-core has been implemented in many tools, like Lighthouse or Microsoft’s Accessibility Insights. Even EmberJS, the open-source javascript framework that I work on, has a library called ember-a11y-testing that can be integrated into an Ember app for automated accessibility testing.

Slide 32

Slide 32

For static analysis, developers can use ember-template-lint, jsx-a11y, and lit-a11y…even vue-a11y is coming along. Each have varied levels of support.

Slide 33

Slide 33

I want to talk a little bit about that. They all mostly work the same, but I’ll talk through it by talking about ember-template-lint because that the one I’m most familiar with and have been working on for a few years now. Here’s some of the benefits: Developers can get linting feedback while they are writing their code It also supports a plugin system that allows teams to define and use custom rules Some rules have automatic fixes built right in, and running the ”fix” flag will cleanup all of the auto-fixable issues Finally, it supports sharing configs across projects, so you can ensure that all of your teams are on the same page, which I think is especially really valuable in today’s globally-distributed workforce

Slide 34

Slide 34

So quick story: Before version 3.0 of ember-template-lint, the print-pending flag gave us a way to roll out new rules. you could take a list of current errors and basically make an “ignore these errors” list in the linter- but it relied entirely on teams to be proactive and treat that list as a burndown list, not an “ignore permanently forever” list. Of course, quality engineering teams understood that we didn’t really want them to ignore these errors forever, but it’s easy to forget about something that doesn’t have a deadline.

Slide 35

Slide 35

One approach was to turn a rule off completely until all existing instances of that error were fixed. Of course, this led to a never ending cycle of trying to turn a rule on, finding new bugs, fixing those bugs…and starting all over again This is because by the time you fixed all the errors, new code would appear with those same errors. Not only is this approach time-consuming and costly… it can really disincentivize teams to keep up with best practices.

Slide 36

Slide 36

So we had issues that could go into the lost void of forever Or stuck in a cycle of always almost ready to turn a rule on We needed to do something different.

Slide 37

Slide 37

Enter support for the TODO. Instead of only having the option to set a rule to warning or error, We can now instruct the linter to find all existing instances of a rule being broken, and create a “todo” for each. This improves on the previous approach in a couple of ways. After a period of time, this “todo” turns into a warning And then an error (which acts as a forcing function- the build breaks.

Slide 38

Slide 38

If we look at this from the perspective of the code we already have and the code we will create, we see that we can break the problem into different parts, so we can immediately benefit.

When a new lint rule is released,

  1. existing code will have todos created and teams can plan to fix incrementally.
  2. Any new code is analyzed as the code is written, so no todos have to be created and no extra planning is needed.

Slide 39

Slide 39

Now you might be wondering, okay but can’t teams just create a new todo? And the answer is, yes, of course they can.

This is about empowering developer teams to move faster. This is about giving PMs the right information so they can make the best planning decisions to support their teams. This is about guidance, not control.

Slide 40

Slide 40

On to the third key part of our strategy- plan to measure our progress. Right now, there is no common, shared standard for metrics in accessibility engineering; my vision is to change that. Metrics play an essential part of our strategy. After all, metrics and reporting is how we empower our senior management (and other decision makers) to give us what we need. Things like…time. Things like budget. Things like vocalization of priorities. When we provide ways to measure this work, we are bridging a very important gap.

Slide 41

Slide 41

Just in case you haven’t had to work with metrics yet, or haven’t in a while, let’s review four of the commonly accepted key criteria of quality metrics: Meaningful. They must be connected to the goals and/or strategy of our organization. Controllable. If a metric is not under your control or influence, it is not meaningful to report (even if it is useful to know about). Easy to Access. Metrics should be easy to find, easy to identify and visible to management Actionable. Any metrics that we define MUST have actionable outcomes.

Slide 42

Slide 42

but what do we mean when we say actionable outcomes? well, we can diagnose a problem improve a process set a goal or observe trends that can inform future work

Slide 43

Slide 43

But before I get into the specific metrics that I think we should be measuring, I want to remind us of Goodhart’s Law- our metrics should never themselves become the entire point. We won’t achieve the outcomes we are after (improved accessibility and user experience) if we merely game the system.

There’s a balance to be had; we need to measure things in a business environment, but we need to deliver quality for our users, not only meet a metric or fulfill a WCAG Success Criterion.

Slide 44

Slide 44

First metric: Potential Violation Count. This metric sets the baseline for us. Now, we all know that there are an infinite number of ways that things can go wrong in any given application. But that isn’t what we mean here. With potential violation count, we want to make the unknown problem a known problem, then solve for each. The baseline is the total number of individual ways an application could fail the legal accessibility requirements. but how do we get that itemized number? In WCAG we have success criteria, known techniques, and common failures. There are also location-specific Legal Standards like Section 508 or the Americans with Disabilities Act. Of course, all of these potential violations represent massive effort that has already been completed by user researchers but how do you get that information in a practical way?

Slide 45

Slide 45

We do this by making an itemized list. There will, of course, be some overlap, but this work is worth doing because it gives us the peace of mind that we really know the edges of this problem. We’re turning an area of ambiguity into an area of clarity We’re making the unknown edges of the problem known and that can help give us confidence.

Slide 46

Slide 46

Before you think “oh my gosh that’s a lot of work” – let me share an effort that has already begun to itemize potential violations. It’s an open source project called the a11y automation tracker, and intends to compile each one of these itemized potential violations with details.

Slide 47

Slide 47

There is an overall detail list, but there is also the ability to dive into each potential violation and see the details about how it can currently be tested,

Slide 48

Slide 48

The existing linting rules and the libraries they are in The testing rules and the libraries they are in, and the relevant WCAG Success Criteria, with links do documentation. Of these potential failures, my initial analysis has indicated that about half of them are either already automated or are potentially automatable.

Slide 49

Slide 49

but what about the other half? This means that the rest still require manual testing, until technology sufficiently advances to programmatically determine these errors. For example: how do we codify intent? In a lot of cases, we still have the opportunity to innovate there. But by taking advantage of the automation that IS available to us, we can reduce the need for manual testing by nearly half, and that’s A LOT less manual testing to be done.

Slide 50

Slide 50

Some of you who are already familiar with the WCAG Success Criteria might be wondering, why not just use those? Well, it’s because they cover generalities rather than specifics. For example, WCAG 1.3.1 (Info and Relationships) is a single success criterion but relates to at least 25+ different failure scenarios. Linting rules have to cover one specific failure, catching as many potential implementation syntax errors as possible

Slide 51

Slide 51

By identifying the edges of the potential violation count, we can then determine related metrics:

Violations for which we can provide automated linting for static analysis Violations for which we can provide automated testing for dynamic analysis Violations that require developer-authored tests Violations that require manual testing These metrics are useful if we build tooling! Once we set a baseline we can target improvements.

Slide 52

Slide 52

There are also metrics that we can look at from the accessibility audits we receive.

What is the Total Bug Count? Bug Severity Count – how severe of an impact do each of these bugs have for our users? For example, a nested interactive element, like a link inside of a button, will result in a complete blocker for a user with assistive tech. On the other hand, a main element with the role of main does not harm the user, it’s merely redundant code. Time to fix- how long does it take a team to resolve an issue? This can be a signifier of a process issue within a team or org, or perhaps a signifier that accessibility hasn’t been vocalized as a priority. If we track this metric, we can start to look at potential causes to help make our apps healthier…in a way we’re already familiar with. And finally, let’s measure violation frequency. In most modern apps, we use components, so maybe we can track what components need to be updated. Or maybe we’ll see that a specific team is contributing a violation more frequently than others. There’s an opportunity to level up our engineers!

Slide 53

Slide 53

So, I’ve given us some essential metrics to track but what sorts of things are we hoping to learn? What impact do we expect to have? We should expect, at a minimum, to see trends as a result of the actions we take.

Slide 54

Slide 54

If we work on tooling, we could expect to see an increase in the number of automated linting rules and tests, since we have a better way to identify specific possible violations. We can also expect to see developers writing more accessibility related tests, since we’re providing them more guidance and education.

Slide 55

Slide 55

we can also expect to see a decrease in support requests for accessibility experts within the org, because developers will have more linting and testing coverage, and a greater awareness of accessibility in general. This awareness should lead to fewer a11y-related issues in new code (specifically, issues we can’t write an automated test for yet) And since we’ve turned automation on, we should see fewer issues in code that has the new automation applied to it

Slide 56

Slide 56

Our audit-related metrics could also inform process from a business/legal perspective, as they can be used to help quantify risk of legal action. When the risk is quantified, we can then quantify the reduction in that risk by taking specific steps to remedy the issues; while this may be a business justification, the result is that our users have an improved member experience, which seems like a win-win. Improved monitoring means that development teams can know sooner if a product’s conformance deteriorates at a rate outside of acceptable thresholds. (Conformance Deterioration Rate (CDR)). It can help us determine a threshold for new learning opportunities. If a high percentage of developers keep writing non-conformant code in a specific way, we can produce materials that will help them on their learning journey We can also use these trends to determine an app’s overall accessibility health. Maybe even put that in a….oh what’s that thing people really love to use at work? Reporting dashboards?

Slide 57

Slide 57

So…a reporting dashboard. Let’s imagine that.

Slide 58

Slide 58

Imagine being able to check in on a product to see how it’s doing and get all of that information easily and quickly.

It could be used to inform decisions: rationalize time, budget, etc. It could be used to demonstrate that an investment in accessibility tooling for designers and engineers could make a big difference. It could be used by sales teams when a potential client wants to know about our app’s accessibility conformance Frankly, there are a lot of use cases but they all come down to this: everyone will be more informed about the decisions they make.

Slide 59

Slide 59

In addition to trend analysis, we can use the metrics we gather to inform future work. Like, Which potential violations currently require manual testing but could reasonably be automated? Just hasn’t been done yet? What violations happen the most? Is there something we could create to make that problem go away or make it easier? How could we make it faster for developers to fix the issues that seem to take a long time to resolve? Is it a code problem? Is it a tooling challenge? Is it a problematic process?

I think there could reasonably be additional metrics to be considered here, but the ones we’ve discussed give us a solid start to quantifying accessibility in our engineering practice, bringing us closer to continuous accessibility.

Slide 60

Slide 60

I think the single biggest thing that we can give not only ourselves but also our senior management is information. The information they need to support us from a business perspective.

The thing is, accessibility IS a civil right, and it is also an emotional issue. And that seems like it’s at odds with business IN GENERAL. But I envision an environment where it doesn’t have to be.

We can accept, as a company value, that accessibility is a facet of quality. We can also rationalize the work that needs doing…with numbers. Just like every other part of the business.

Maybe in a perfect world, we wouldn’t have to do these things, but also…we CAN do this, so why not give business what it needs? After all, the outcomes are what matters.

Slide 61

Slide 61

You can use sites like Please fund a-eleven-y dot com a website that provides a list of the known areas where funding or support would improve the state of accessibility across the web. Maybe your company has engineer time that can be used to help solve some of these issues. Maybe your company has funding that can be allocated to hire technical writers to improve the state of accessibility documentation.

Slide 62

Slide 62

Or, perhaps you can contribute to some of the open-source projects that exist to provide linting rules for the framework your company uses.

Slide 63

Slide 63

Another potential outcome is to develop training materials that help relate the information that people already know to accessibility concepts that they maybe don’t, like this training project that I’ve been working on at HashiCorp to help relate existing design heuristics with accessibility concepts

Slide 64

Slide 64

There are so many possible outcomes here…it’s like a choose your own adventure! There is room for everyone to contribute.

Slide 65

Slide 65

At the beginning of this talk, I made a few promises. So let’s see if I kept them. As we have looked at how to think about our code and what metrics to consider, I hope you have seen the path forward for continuous accessibility. I hope I have inspired you to create something new or contribute to an effort already in progress. I hope I have empowered you to think about accessibility in your design or code or even in your process in a new way.

And if you’re a decision maker, I hope I have shown you that continuous accessibility is the thing you’ve been waiting for. Think SRE, but A11y.

Slide 66

Slide 66

Continuous accessibility is good for business, but it’s great for users. And that’s what we’re really going for.

Slide 67

Slide 67

Thank you!!