A presentation at EmberConf 2021 by Melanie Sumner
Hello! I’m Melanie Sumner, a disabled military veteran turned software engineer and I’m delighted to speak with you all today. I wish it could be in person but I am looking forward to next year! I’m also a member of the framework core team, and the WAI-ARIA working group and I do these things as part of my job as as senior software engineer at LinkedIn, where I work on digital accessibility solutions that scale.
Since many of you will be familiar with my usual style of talk - This talk is different. In my other talks, I’ve spoken about what accessibility is and things that we can do now
If you are interested, you can find my previous talks by visiting my speaking profile on notist. But this talk is different because it is about my vision for how I want the world to work and how I think we can get there.
And, Just in case this is the first time you’re hearing about accessibility, let’s run through a few common terms, since over the next couple of days you are likely to hear other people talking about accessibility, too. When we say “accessibility”, we are specifically focused making inclusive website so people with disabilities can use what we create. You might also hear it referred to as A eleven y, which is a shortened abbreviationthere are eleven letters between the A and Y in the word accessibility. I’ll also refer to the web content accessibility guidelines, or WCAG for short, as the standard by which our applications are evaluated through the use of the success criteria they provide. Assistive Technology includes hardware or software that people with disabilities use to access the web or use mobile devices. For example, a person who is blind may use screen reader software to browse and use websites on the internet through desktop browsers or mobile devices.
So Today I am going to share with you my vision, a vision I am calling continuous accessibility.
Let’s jump right in.
I want you to take a deep breath and let yourself imagine. Imagine having greater confidence in the accessibility of your code.
I want you to imagine more easily delivering accessible experiences at scale
Finally, I want you to Imagine not being afraid of losing customers or facing expensive lawsuits due to accessibility blockers on your site.
In software engineering today, we have continuous delivery
we also have continuous integration And these two things have drastically improved our lives. Unless, of course, you’re the engineer working on bugs for these two things, in which case you probably swear a lot, but the rest of us are grateful to you , so just know that we appreciate your sacrifice.
But now I want us to turn our thoughts toward a new idea- continuous accessibility. How will we get there? How can we deliberately achieve this vision? We have to be on purpose.
The well-established principles of continuous software engineering remind us to build quality in
To work in small batches
To let computers do the boring stuff so humans can do the hard stuff
To always be improving
And that everyone is responsible for doing these things.
In this talk, you will learn the essential points of strategy for continuous accessibility.
What does that strategy look like? well, this three-part strategy includes
A plan for the code we already have
A plan for the code we will have in the future
And a plan for how to measure our work, and we will discuss why that is so important.
But before we dive into that, I’m excited to share with you that we have already started this continuous accessibility journey with the newest version of Ember Template Lint. As of the latest version, no longer in beta, we deliver template linting in such a way that It ensures our existing code has a path to become more accessible, and our future code will be more accessible by default We have a path forward that is compassionate for both developers AND our users. And that’s really exciting to me
At LinkedIn, we have manually rolled out this new feature to some of our apps already in production We also took what we learned through the manual implementation process, and built automated tools to do this work in the future. And we are also building trackers to ensure that the configurations remain correct over the lifetime of the codebase.
Any plans we make to improve accessibility in our products should include plans for the code we already have. We need to think about the age of our code base. We need to think about how we plan for library upgrades, because it’s cool that we improved accessibility on that popular UI addon, but what does this mean for existing users? What does the upgrade path look like? We need to think about how we will deliver new features or new developer tools. The code that already exists will not be able to consume the latest upgrades unless we’ve done this thoughtfully and purposefully. and it will likely take some trial and error :)
This is especially true for apps at scale. It can be tricky to update a dependency especially when it includes breaking changes. Depending on the size of your codebase Taking in a new version of a dependency can mean extra developer coordination about which new features to use There are product and business priorities to consider Features we create for accessibility automation should makes it simpler to deliver improved products and tooling to support accessibility as a facet of our craft.
For the code we have now, we can wait for our users to report issues to us- but at the risk of losing customers.
We can also rely on audits to tell us where the issues are.
Ideally, we are automating our code linting and tests, periodically checking the code we already have. This gives us higher confidence that as technology progresses, our sites still work as expected. Of course, there is still a lot of improvement we can bring to this area.
But this approach allows automation to be our first line of defense, backed up by accessibility audits and user reports (because users will still tell us when they run into blockers)
We need to plan for the code we will write in the future, or we will be much more likely to make the same mistakes of our past. Who knows what comes next? The way we wrote code ten years ago is different from the way we write code today. what does our strategy need to include to prepare us for these unknown unknowns?
Of course, We will continue to follow the principles of continuous software engineering. One of these, letting computers perform repetitive tasks so people can solve hard problems, is what I want to zero in on. This is something that we have- and will continue to- improve on. Let’s take a look at the way automation in accessibility can help.
Today, developers have access to automated testing through the axe-core library that can be integrated into the continuous integration and delivery mechanisms that are already used to test and deliver our code. We have a lot of options, too! We can use Lighthouse, or Microsoft’s accessibility insights, or the ember-a11y-testing addon, or even the axe-core library itself.
For static analysis, we can use ember-template-lint. Now I will note here that eslint is also available for static analysis, but it does not currently contain any support for accessibility.
Developers can get linting feedback while they are writing their code It also supports a plugin system that allows teams to define and use custom rules Some rules have automatic fixes built right in, and running the ”fix” flag will cleanup all of the auto-fixable issues Finally, it supports sharing configs across projects, so you can ensure that all of your teams are on the same page. 35
Before version 3.0 of ember-template-lint, the print-pending flag gave us a way to roll out new rules. you could take a list of current errors and we could basically make an “ignore these errors” list in the linter- but it relied entirely on teams to be proactive and treat that list as a burndown list, not an ignore permanently forever list. Of course, teams understood that we didn’t really want to ignore these forever, but it’s easy to forget something that doesn’t have a deadline.
One approach was to turn a rule off completely until all existing instances of that error were fixed. Of course, this led to a never ending cycle of trying to turn a rule on, finding new bugs, fixing those bugs…and starting all over again This is because by the time you fixed all the errors, new code would appear with those same errors. This is time-consuming, and costly- which can really disincentivize teams to keep up with best practices.
So we had issues that could go into the lost void of forever Or stuck in a cycle of always almost ready to turn a rule on We needed to do something different.
Enter the todo In the latest versions of Ember template lint, we now have the todo.
Instead of only having the option to set a rule to warning or error, We can now instruct the linter to find all existing instances of a rule being broken, and create a “todo” for each. After a period of time, this “todo” turns into a warning And then an error
Let’s look at this from a different perspective, from the perspective of the code we already have and the code we will creat.e When a new lint rule is released, existing code will have todos created and teams can plan to fix incrementally. For new code, it is linted as the code is written, so no todos have to be created and no extra planning is needed.
And this feature is intended to mirrors the qunit todo feature, which gives it better consistency for developers who already use the feature. And you can try it out yourself, today! once you’ve installed the most recent version of the linter, you can run the linter and create todos with the update-todo flag you can see the todo count when you run the linter with the include-todo flag and once an issue is fixed, you can automatically remove the todo with the fix flag.
It’s so incredible that we can try this now, and even more incredible that it is built into all new ember apps by default. Just one of the many reasons I love using Ember. so now let’s turn our attention to the third piece to this puzzle, metrics. We need a way to quantify our progress.
Right now, there is no common, shared standard for metrics in accessibility engineering My vision is to change that. Metrics play an essential part of our strategy. After all, our goal is to demonstrate the business value of accessibility and make it a default part of our engineering practice.
Just in case you haven’t had to work with metrics yet, let’s review four of the commonly accepted key criteria of quality metrics: (cite the authority here) Meaningful. They must be connected to the goals and/or strategy of our organization. Controllable. If a metric is not under your control or influence, it is not meaningful to report (even if it is useful to know about). Easy to Access. Metrics should be easy to find, easy to identify and visible to management Actionable. All metrics that we define MUST have actionable outcomes.
but what do we mean when we say actionable outcomes? well, we can diagnose a problem improve a process set a goal or observe trends that can inform future work
But before I get into the specific metrics that I think we should be measuring, I want to remind us of Goodhart’s Law- our metrics should never themselves become the entire point. We won’t reach our goal if we game the system.
Now, we all know that there are a nearly infinite number of ways that things can go wrong in an application. But that isn’t what we mean here. With potential violation count, we want make the unknown problem a known problem, then solve for each. The baseline is the total number of individual ways an application could fail the legal accessibility requirements. but how do we get that itemized number? Web Standards WCAG (success criteria, known techniques, common failures) Location-specific Legal Standards for example in America we have Section 508, ADA
and there are different standards for the european union and canada and different countries around the world Failures identified by audit findings of course all of these potential violations represent massive effort that has already been completed by user researchers but how do you get that information in a practical way?
We do this by making an itemized list. There will, of course, be some overlap, but this work is worth doing because it gives us the peace of mind that we really know the edges of this problem. We’re turning an area of ambiguity into an area of clarity We’re making the unknown edges of the problem known and that can help give us confidence.
Before you think “oh my gosh that’s a lot of work” – let me share the effort that has already begun to itemize potential failures. It’s an open source project called the a11y automation tracker, and intends to compile each one of these itemized details.There is an overall detail list, but there is also the ability to dive into each potential violation and see the details about how it can currently be tested, the possibilities for the future, and the relevant criteria, with links do documentation. Of these potential failures, my initial analysis has indicated that about half of them are either already automated or are potentially automatable. but what about the other half? This means that the rest still require manual testing, until technology sufficiently advances to programmatically determine these errors.
And I want to emphasize this again- you still need manual accessibility testers IF you’re a small company, just remember that accessibility is a journey. Do what you can now, and do more as you’re able. If you’re a large company, I want to encourage you to NOT outsourcing this work- In my experience, a few well-trained testers that share your company’s values will be much more resource efficient than an entire army of outsourced testers. No matter what size company you are, I strongly encourage you to make manual testing an essential part of your accessibility strategy
Some of you who are already familiar with the WCAG Success Criteria might be wondering, why not just use those? Well, it’s because they cover generalities rather than specifics. For example, WCAG 1.3.1 (Info and Relationships) is a single success criterion but relates to at least 25+ different failure scenarios. When I write a linting rule, it needs to cover one specific failure (even if that failure can present in different kinds of syntax).
By identifying the edges of the potential violation count, we can then determine several related metrics: Violations for which we can provide automated linting for static analysis Violations for which we can provide automated testing for dynamic analysis Violations that require developer-authored tests Violations that require manual testing
There are also metrics that we can look at from the accessibility audits we receive. What is the Total Bug Count? (TBC)? Bug Severity Count (ISC) – how severe of an impact do these bugs have for our users? Time to fix- how long does it take a team to resolve an issue? And violation frequency
So, I’ve given us some essential metrics to track but what sorts of things are we hoping to learn? What impact do we expect to have? We should expect, at a minimum, to see trends as a result of the actions we take.
we can expect to see an increase in the number of automated linting rules and tests, since we have a better way to identify specific possible violations.
we can also expect to see a decrease in support requests in a11y-related issues in new code and fewer issues in code that has the new automation applied to it
Our audit-related metrics could also inform process from a business/legal perspective, as they can be used to help quantify risk of legal action. When the risk is quantified, we can then quantify the reduction in that risk by taking specific steps to remedy the issues; while this may be a business justification, the result is that our users have an improved member experience. Improved monitoring means that development teams know sooner if a product’s digital a11y conformance deteriorates at a rapid rate (Conformance Deterioration Rate (CDR)). it can help us determine a threshold for new learning opportunities. If a high percentage of developers keep writing non-conformant code in a specific way, we can produce materials that will help them on their learning journey we can also use these trends to determine an app’s overall accessibility health. Maybe even put that in a….oh what’s that thing Ember is so good at? A dashboard?
In addition to trend analysis, we can use the metrics we gather to inform future work. Like, Which possible violations currently require manual testing but could reasonably be automated? What violations happen the most? Is there something we could create to make that problem go away? How could we make it faster for developers to fix the issues that seem to take a long time to resolve? There are additional metrics to be considered here, but the ones we’ve discussed give us a solid start to quantifying accessibility in our engineering practice, bringing us closer to continuous accessibility.
I think there’s a new tool that our community could create I have a vision of this being available as an open-source tool, something that anyone in the community can use to improve the measurements of accessibility in their code base. Of course, this slide is a little bit ugly but if you’re offended by the ugly design please consider helping out to make this better . With our metrics in hand, We could use this data to help us clearly and quickly see how we’re doing.
As we have looked at how to think about our code and what metrics to consider, I hope you have seen a path forward toward the vision of continuous accessibility. I hope I have inspired you to create something new or contribute to an effort already in progress. I hope I have empowered you to think about accessibility in your code in a new way.
A lot of the strategy I have talked about today is of my own creation, but I am part of a team and stand on the shoulders of giants. And I’d like to take a moment and recognize the contributions of others to this project. Robert, among many many other things, maintains ETL and directed the idea to add the todo feature to ember-template-lint. My colleague Steve Calvert drove the todo implementation and designed the decay days feature. Internal teams at LinkedIn have been enthusiastic early adopters; it really allowed me to work out the kinks of our strategy to implement template linting at scale. And of course, my husband Joseph who supports me and my career and is the reason I can take the time to speak to you today. as a research scientist, he has taught me to think more like a research scientist, and I am really grateful for that.
In closing, I’d like to say this. This talk has been focused around providing a strategy to achieve continuous accessibility. But when a measure becomes a target, it ceases to be a good measure. I want you to remember that.
Accessibility is for everyone.
How do you balance the challenge of continuously delivering features and products that are inclusive and accessible? In this talk, we will examine the idea of continuous accessibility- my vision for the future of accessibility in engineering practice. We will discuss a three-part strategy for how our community can make this idea happen, starting this crucial conversation for more inclusive products and elevated artisanship.
The following resources were mentioned during the presentation or are useful additional information.
The JavaScript framework
The site that tracks available accessibility-related automation for linting and testing.
The ember-template-lint library, a linter for your template code.
Here’s what was said about this presentation on social media.
