Be the Villain

• Be the Villain: Subverted Design
• AXE-CON 2021

• Hey, it’s the internet! Heck yeah!

• We’ve got web surfing!

• Cool dogs!

• Cat memes!

• Chatting with your friends!

• Watching videos!

• And whatever this is!

• Everything is awesome when you’re online!

• Or is it?

• Before we get started, there are a two things I’d like to cover
• First is a content warning for this talk

• The nature of this talk’s content means I will be discussing some pretty terrible things
• The list is:
• Domestic abuse, drug abuse, genocide, harassment, racism, stalking school shooting, and swatting

• I will only be touching on the concepts described in this list lightly to prove a point, with the exception of school shootings, genocide, and harassment, in that order
• I’ll need to provide more detail for these three topics in order to better articulate points I want to make about how to avoid them
• If this talk’s content is something you don’t think you’re in a good place to listen to, I completely understand
• The talks by Preety, Plum and Jack, and Wilco and Mary Jo are also happening at this time slot, and they both look great
• I’ll also be posting this talk, including annotations and links, after it is concluded, if you feel like this is something you want to consume at your own pace

• The second thing I’d like to call attention to is that I’m not the first person to be discussing the concept of subverted design, and hopefully not the last
• A lot of the research for this talk is built from the painstakingly-assembled information created by women working in our industry, or web-adjacent technology industries

• I would also like to say that finding people who have discussed this topic was difficult
• I attribute part of that to survivor bias, the logical error that teaches us that we overlook what isn’t present as part of our selection process
• It is my belief that many who could talk about this are no longer here, having left, or having been forced out of the industry

• The goal of today’s talk is to equip you, the audience, on how to identify subverted design and guard against it
• The beats I’ll be hitting on for today’s talk are:
• What isn’t subverted design
• How subverted design can ruin a product
• Subverted design in the headlines
• How subverted design is our responsibility, and
• How to identify where subverted design could occur

• I might be biased, but I think Inclusive Design is a really great framework for a lot of design-related things
• The reason for this is that the nature of inclusivity requires thinking about how people can get left out and be discriminated against
• Things like equality, accessibility, and fairness are really important, but we need to acknowledge the reason Inclusive Design even exists in the first place is because we oftentimes accidentally, or intentionally create the opposite
• As a methodology, Inclusive Design allows us to take an idea and see how we can facilitate, or hinder access And this talk is very much in the spirit of that

• So now let’s describe what subverted design is not
• Defining these more commonly-understood parameters helps to give us a framework for what does, and what does not qualify as subverted design

• First off, subverted design is not bad design
• On this slide is a web page with a bright rainbow background and a marquee banner that reads Accept Jesus Forever Forgiven church
• It inexplicably features: a passage from Romans 6:23, a background MIDI, a hot air balloon, a World War II fighter plane, a dove, some arches, a step ladder, and a banner ad for Maverick Christians.com
• This prismatic vomit is a rather extreme example, but it is honestly pretty direct about its intentions, and there is little room for doing anything other than the six or so calls to action it wants you to undertake

• Subverted design is also not what I like to call Beholden Design
• Remember in the Before Times where we could go to the theater and spend our hard-earned money to watch a movie?
• And that how, despite paying for this service, you would still be forced to watch some advertisements before the thing you actually wanted to watch was shown to you?
• That’s an example of Beholden Design
• It’s where a person enters into a system and is forced to endure something that benefits the system to ultimately get what they want

• Speaking of systems, subverted design isn’t hacking
• This is a little more subtle than the previous examples
• Hacking is where someone breaks a system on a deeper level, typically by finding an exploit in code and using that exploit to gain access and exert control on the underlying systems that power the experience

• Subverted design also isn’t a Dark Pattern, although they’re close cousins
• If you’re not familiar, a Dark Pattern is a technique where a website or app deliberately uses a confusing user experience to trick you into doing things you didn’t intend to

• For the example displayed here, SportsDirect has automatically added a product to your shopping cart and charges you for it, hoping that you won’t notice when you check out
• Gross

• So now you might be thinking that this is just a one-off, and that I’m really grasping at straws here
• What I say back is that we should not be so quick to underestimate what people are capable of doing with the resources we give them
• In design, there is the bias that our users are simple, and require straightforward instruction, oftentimes to the point where we infantilize them

• What I am telling you is that even a less digitally-literate user may be able to subvert a system, provided there is enough motivation on their part
• Here is an example of people figuring out that they could get free burgers from a McDonalds self-serve kiosk by customizing an order to a ridiculous degree -It’s a funny example, but the real fault here is the designers and developers who didn’t anticipate the logic behind their rules for order customization being taken to an extreme.

• So what exactly is subverted design, then?
• Now that we have some solid examples of what it isn’t, I’d like to tell you a story that illustrates an example of what it could be
• This isn’t something I’ve made up, it was a potential client we were considering working with at my job
• What I’m going to discuss here is how a poorly-thought out system can be taken advantage of by someone with malicious intentions
• To do this, we’ll be breaking this system down into discrete components, to show how the components relate and rely on each other, and where that can fall apart under stress

• This is a school
• Schools are effectively a collection of classrooms, each classroom containing a teacher and their students

• In a school’s organization, there is admin staff that the teachers answer to.
• This is a simplified version of things, but for the sake of this talk, let’s say this is the principal

• This is a school shooter
• If you are not from the United States, this is a horrifically common thing that we’ve normalized
• A school shooter could be a student, or an adult either working in the school, or who enters into it

• In a school shooting scenario, a shooter may target a classroom
• Nearby teachers will become aware of this once the shooting starts

• These teachers will reach out to emergency response services, typically via a text or phone call from their smartphone
• Because again, this is America, and we’ve normalized this horror to the point where we need to train our teachers for this kind of formerly unthinkable scenario

• Each emergency response service can then take this reported information and push it out to school staff, including other teachers in the school, as well as other associated emergency response services
• This allows both teachers and school admin, as well as emergency response services, to become aware of the situation, and act on information as the crisis unfolds

• However, one big problem is that it is this is real life, and real life is messy
• Things aren’t as straightforward as the scenario I just described
• In addition to reaching out to emergency services, teachers will reach out to family members, fellow coworkers, friends, and loved ones
• And to add to this complexity, emergency services are constantly communicating with each other as new information is made known

• Friends, family, and loved ones may also try to reach out to teachers, students, and school admin when they learn about what is happening
• In addition, local and national media may also try to butt in
• This creates a ton of chaos in an already chaotic, high stakes situation
• With it comes increased chances for out-of-date information, or bad information to be disseminated

• To add to this, too much communication happening too quickly in a small area can cause the whole thing to crash

• To address this issue, the potential client had the idea that they could focus their efforts on a single individual

• In this case, it would be the principal, or other high-level school official

• In the event of a shooting, teachers would be instructed to not pass information along to emergency services, and instead all notify a central figure

• The idea being that the central figure could communicate with emergency officials on their behalf

• Emergency services would then coordinate with each other, as well as disseminate information back to the school in one single consolidated transmission
• In theory, this would streamline communication to keep bad information down, as well as helping to keep the network up
• In practice, however, they failed to think like a villain

• An attacker who gets control of a teacher’s phone now has a one-stop shop for accessing information about other teachers, students, and emergency services
• It lets them know all sorts of information that would make an already terrible situation worse

• If they get control of the school administrator’s phone, it gets even more dangerous
• Using the client’s proposed feature set, they could send messaging to both emergency services and school staff
• Messages that could, for example, corral staff, declare false alerts and all clears, and facilitate taunting, goading, and other forms of psychological manipulation

• It also failed to anticipate a scenario where there were multiple coordinated attackers, a situation that has tragically happened before on multiple occasions
• After a review of the full proposal, we were able to determine that a school shooter could potentially get access to:

• School floor plan

• Teacher names

• Teacher locations

• Teacher status

• Student names

• Student locations

• Student status

• Incoming police

• Incoming emergency services

• And messaging

• Identifying the various ways this product could be subverted translated into the very real risk for harm to be amplified
• This discovery represented a hard stop on the progress of a product that had already had tens, if not hundreds of thousands of dollars of development time poured into it
• On this slide is a panel from a golden age comic depicting a sportscar that’s crashed into a tree placed at a fork in the road

-= The school shooting reporting app is one example, but this subversion of technology for personal purposes has been happening for quite some time now, and will continue to happen

• If you start to look for this kind of thing, you will see it everywhere

• In The Washington Post:
• Women face new form of harassment on metro: Airdrop

• In The New York Times:
• Thermostats, locks and lights: digital tools of domestic abuse

• In NewsHub:
• Auckland woman ‘creeped out’ after restaurant worker uses her contact tracing details to hit on her

• In The Verge:
• Twitter bans animated png files after online attackers targeted users with epilepsy

• In The Daily Dot:
• Facebook is helping husbands ‘brainwash’ their wives with targeted ads

• In Buzzfeed News:
• Instagram’s opioid recovery hashtags are full of drug dealers

• In The Verge:
• Swatting over Call of Duty game results in deadly police shooting of Kansas man

• I keep a list of these headlines as I run across them

• And part of creating this talk was whittling this list down to a length that could prove my point

• Without running over time

• Or completely shutting you, the audience, down
• But I do need to stress that this is happening, and it is widespread
• As technology enters into more and more of our daily lives, there are more and more chances for this subversion happen
• It also means the scope of harm gets exponentially worse
• Culminating in this:

• Mynmar military personnel used Facebook’s features to orchestrate an ethnic cleansing of an minoritized population
• This is genocide, facilitated by technological subversion.
• Don’t look away

• We need to stop being naive or this will continue to happen

• Design finally got a seat at the table, and you know what? That’s great
• However, this also means the scope of our job has expanded
• We’re no longer people who just make pretty things, we now help shape how businesses operate in the world

• With this new set of responsibilities comes the uncomfortable understanding that we may not be the heroes we thought we were
• Historically there’s been this notion that we fight for the user’s best interests and be the first line of defense for guarding them online
• But realistically, design is now mainly a profession that uses aesthetic and usability best practices to help businesses more efficiently generate profit
• This shift is worth acknowledging for many reasons, but the one I’d like to focus on for this talk is the vacuum that this creates
• And how one of the many things that can enter into this gap is a higher likelihood for harm
• On this slide is a comic panel depicting an ecstatic man in a suit half-submerged in a bag of money, completely oblivious to the sinister hand reaching for him

• We need to understand that we’re still pretty new at thinking about how to prevent harm, but others in the tech industry have been doing it for a long time
• We can look to the field of information security for example, which has been practicing threat modeling for quite some time now

• Threat modeling is the process of identifying all the parts that make up a system…

• …and how they relate to each other

• With the knowledge of what the parts that make up the system are, and how they operate,

• We are then able to compartmentalize them, and control who has access to what
• This proactively limits the amount of harm an attacker can inflict on a system

• Any feature can be abused.
• The gist of this talk is wisdom from Kat Fukui, Senior product designer for communities at GitHub
• I greatly admire, and have benefitted from Kat’s work, and you’d be smart to read more of what she’s written

• So, we now know that harm happens because of a naive view of technology, that people can exploit systems if sufficiently motivated, and that we need to consider both the systems we work on, and the component pieces they’re made from

• In order to fight monsters, we must become monsters
• It’s time to be the villain
• This is our origin moment, our tragic backstory,
• Our chance to seize control and show the world what we’re made of

• Threat modeling is great starting point for thinking about how to prevent subverted design
• But we’re villains now. Resourceful, relentless, radical
• We need to formulate bigger, bolder plans

• I think that turning to the Inclusive Design principles is a great way to start
• If you are unfamiliar, they are seven principles that, if followed, help to make accessible digital experiences
• Remember: It’s not all sunshine and rainbows and kittens, it’s a methodology to take an idea and see how it can facilitate, or hinder access

• The first Inclusive Design principle of Providing a Comparable Experience teaches us that content can be accessed in multiple ways
• That people can accomplish tasks in a way that suits their needs without undermining the quality of the content
• Examples of this include alternate image content and synchronized captions

• As villains, we can flip this
• My first flipped Inclusive Design Principle is to Force a Single Path
• This opens up opportunities for exploitation
• For example, needing to rely on someone else to read the content for you requires a huge extension of trust
• Think banking, medical records, mental health services, legal information, and all other kinds sensitive data
• What happens when this kind of trust gets broken?

• The second Inclusive Design Principle of Considering Situation tells us to ensure our content works in a variety of scenarios:
• First time and experienced users, outdoors and indoors, wifi and a metered data plan, bright light and low light, etc.

• Flipping this, my second inverted principle is to Ignore Circumstance
• Demand specific conditions to work, and restrict access until they’re met,
• An example of this is requiring someone to be in a specific location in order to work
• In controlling the conditions, the system can control how its users behave
• This is predatory, with the side-effect of making users more predictable, which in turn makes them more exploitable

• The third Inclusive Design principle is to Be Consistent,
• To use familiar conventions and apply them consistently
• Examples of this are using buttons that look like the buttons on other websites, and links that look like the links on other websites, then making sure they appear consistently on your own site

• When I invert it, we Act Capriciously and create sudden, unannounced and unanticipated changes in how our experience works
• This allows a bad actor to easily trick and trap the unsuspecting
• For example, tricking people into dropping out of queues, misdirecting payment, or sending shock content

• Giving Control places people’s needs first, and ensures that they are able to interact with content in their preferred way
• Examples of this include the ability to scroll via keyboard, stop animation, and zoom in and out of content

• The opposite would be Removing Autonomy
• Here we downplay voluntary interaction, doing things like ignoring expressed preferences or automatically triggering actions without consent

• The fifth Inclusive Design principle is to Offer Choice
• This is providing different ways for people to complete tasks, especially tasks that are complex or non-standard
• This principle is reflected with things like individual and bulk actions, expanded and condensed data views, and graph and chart views of tabular data

• Its opposite would be to Limit Options, forcing seemingly-arbitrary conditions on your users
• Here, you restrict things to a single, narrow view, and user flows that make no external sense

• The sixth Inclusive Design principle is to Prioritize Content, to help users focus on core tasks, features, and information
• Here, you do things like place the main information prominently, anticipate information before it is needed, and ensure content loads quickly

• When I invert it, it becomes Obfuscate Purpose, the act of misdirecting and downplaying the actual goals and consequences of action
• Examples of this are hiding fees, legal gotchas, and other schemes

• The final Inclusive Design Principle is to Add Value.
• This is the act of weighing the positive impact of new and existing features, and how they interrelate
• An example of this from the web platform is how new features such as CSS grid can greatly enhance a layout, but still gracefully degrade into a perfectly readable single-column layout for browsers that don’t support the feature

• The flipped version of this final principle is to Fill with Gibberish
• Hide important and consequential information behind complicated language, jargon, distracting decoration, and the like
• A really good example of this is trying to buy a car

• To recap, my 7 inverted Inclusive Design principles are:
• Force a Single Path, Ignore Circumstance, Act Capriciously,
• Remove Autonomy, Limit Options, Obfuscate Purpose, and Fill with Gibberish
• Each of the seven inverted Inclusive Design Principles I just described weaken a quality digital experience, and each one presents an opportunity for someone to capitalize on a system to cause harm
• Combined, they create a system that is ripe for for subversion
• And as villains, we’d do good to remember this when taking another look at the products we work on

• So let’s take this out of the abstract and work through an example of how we can identify areas where bad behavior could occur,
• And importantly, what can be done about it

• It’s a joke and the joke is that it is the truth that on a long enough timeline every product eventually gets a messaging service
• In this example, Christina is using a chat widget to ask for help with her service on a cable provider website

• The other, sadder joke is that on a long enough timeline messaging services becomes dating apps
• Here, the responding agent says “you’re hot, can I call you?”
• While he can’t see Christina, he can access her account, which includes information such as her address
• He can also search around and find her social media accounts, where she works, and possibly even where she likes to hang out
• As to why, maybe he’s bored, or angry, or desparate
• You can never know.
• But we can know that in this scenario, he’s abusing position, power, and circumstance to put Christina in an incredibly uncomfortable position

• When we consider a simplified version of this chat support widget’s user flow, only the happy path is considered:
• A customer asks a question and either it is answered, or clarifying information is requested and delivered, leading to an eventual resolution of the issue

• However, when we think like a villain, we can identify areas where both the agent and the customer can act inappropriately, and what can be done about it
• It’s important to note here that issue resolution may not be the end goal in this circumstance, and may in fact never come into play
• A successful user flow in this context can be where harm is removed

• All of this translates into user interface decisions

• Adding additional functionality for secondary actions allows us to empower the chat widget’s users

• Reporting, blocking, and exporting all can help prevent and minimize harm:
• Reporting logs the incident and notifies someone charged with reviewing the reports it generates
• Blocking breaks contact and programatically prevents this agent from being assigned to this customer in the future, and
• Exporting provides an artifact for the user that can persist after the session is ended and the browser tab has been closed

• Note that each one of these features represents additional work, especially if things aren’t set up on the backend to accommodate these new changes,
• So it is important to identify and codify these features as early in the process as you can
• On this slide is a comic panel featuring astronauts frantically repairing a silver rocketship that is already in-flight

• Another really important thing you can do is slowly, gently introduce the idea of subverted design to the organization you work for
• By which I mean tackling subverted design head-on might not work the way you think it will

• People don’t like to hear bad news, especially bad news about things they failed to consider on the things they worked on, and especially bad news about how that might hurt people
• And then you factor in things like institutional culture, pressure from management, and toxic positivity.
• On this slide is an illustration showing an ostrich with its head buried in the sand
• If you don’t ease people into the idea of their work having the potential to create harm, your efforts might blow up in your face
• Here’s a Twitter conversation between designers Aaron Z. Lewis and Scott Perket that talks about just that

• Aaron says, quote:
• In light of the latest FB scandal, here’s my proposal for replacing Design Sprints: Black Mirror Brainstorms
• A workshop in which you create a Black Mirror episode. The plot must revolve around misuse of your team’s product.
• Pair with @brownorama’s idea of “abusability testing.” Endquote.
• Here, Aaron is suggesting a roleplaying exercise where you take an existing feature that people like and try and subvert it
• Scott replies, quote:
• I frequently asked genuinely earnest questions of teams along these lines while working on ads measurement at Facebook and was considered negative and not a cultural fit.
• I did so from a stance of experience. Reminder: blind optimism usually leads to bad outcomes. Endquote
• With his reply, Scott warns about how this kind of proposal has been received in the real world

• Scott wasn’t wrong despite this alleged negative cultural fit.
• Of its many, many issues Facebook has enabled people subverting its ad platform to discriminate based on age, gender, race, religion, and other characteristics
• Since Facebook has effectively become digital infrastructure, this means it may prevent access at scale to things like jobs, housing, and credit
• This redlining is a repetition of racist segregation policies that occurs in the physical world

• You see, easing people into the idea of subverted design is a matter of framing
• On this slide I’ve taken the screenshotted conversation between Aaron and Scott and placed a gigantic ornate golden frame around it

• Heroes charging in full of righteousness and declaring that someone could abuse something will shut things down
• Especially if this accusal is hypothetical
• The act immediately challenges personal and organizational ego
• And in fact, someone in the organization who has decision-making power could even be actively exploiting the services you’re calling attention to
• It’s a bad thought, but remember: we’re the villains now, so we need to think this way

• Instead, gather decision makers in a room
• The more diverse the better, the more experience and perspectives the better
• Once assembled, ask them, “How could we make this feature terrible?”
• It’s a fun exercise. The prompt creates a space where it’s safe to come up with ideas about what bad design is without fear of punishment or retaliation

• Villains are sneaky.
• Start with obvious things like ugly design and bad inputs.
• Then maybe float the idea of poor performance and cryptic error messages

• Slowly nudge them them towards worse and worse things.
• Chances are good they’ll start to move into exploitative behavior territory on their own, but if they don’t, subtly and sneakily suggest it a casual manner to help guide them towards where you want them to go

• With all the suggestions written down on sticky notes, have them sort the ideas into groups, and then have them label the groups

• And tada! Subversion is now one of a few concerns you’ve all worked together to uncover

• Now the dynamic has shifted
• You—the manipulative supervillain puppeteer that you are—have just created a superhero team
• This new team has defined what evil is, and now knows what that evil will look like—including subversion
• With this newfound knowledge the superhero team will now rally around protecting their users and feel good about doing so, because now they’re the ones feeling empowered to set things right
• On this slide is a part of a golden age comic showing a 5 man superhero team who have very serious-looking expressions, but who are also ridiculously-dressed

• Another really important thing you can do is create a code of conduct for your companies, teams, and projects

• Codes of Conduct do a really important job: they outline expectations of what is, and what is not acceptable behavior
• This creates a shared understanding of community norms
• It also outlines responsibilities, and provides benchmarks for behavioral and ethical standards, as well as methods to report bad behavior in a safe and secure way

• On this slide is Multiple Man fighting some superheroes.
• Multiple Man is a supervillain that has the ability to instantly create an infinite amount of copies of himself

• The Contributor Covenant Code of Conduct is a code of conduct you’ll frequently see on GitHub and many digital projects
• The Covenant itself makes for a good starting point, but the problem is it is presented as a turnkey solution
• You can click a button to drop a copy of the Covenant into a project without really knowing what it fully is or why it covers what it covers
• Each community has its own specific needs, and the Code of Conduct should be manually reviewed and updated to reflect that

• The other important bit about Codes of Conduct is adopting one isn’t the same as internalizing it
• We all know we don’t read Codes of Conduct, but we really should
• And we should encourage others to do so by showing the behavior we want modeled!

• Harm is also inevitable
• Sadly, it is impossible to predict every possible way a system could be misused
• But we can preemptively set up guidelines about what to do when harm does happen
• For this, you’ll want an Incident Response Guide

• The A11Y Project, a project I proudly help maintain, has one
• Much like a Code of Conduct, it creates a shared set of expectations around what to do if an incident occurs
• We require each maintainer to read through and sign it, as well as read through a sample scenario,
• This helps maintainers understand how abstract guidelines can be applied in a real-life situation.
• And this really isn’t about us as maintainers, either
• It’s about the community we serve, and prioritizing their health and safety in a fair and equitable way

• Speaking of community, one last little bit about villains:
• We don’t have to be miserable islands of fear and resentment
• We can team up
• On this slide is the top half of issue number 2 of Marvel’s Giant-Size Super-villain Team-up, featuring Dr. Doom and the Sub-Mariner

• And on this slide is a villain addressing a room full of other villains from a podium
• Hear my words, fellow villains: Alone, we may struggle in vain,
• Isolated, we may face stigma for daring to speak the truth
• But if we band together, we can leave our mark on the world, and show all those fools who dared doubt us
• You’ve all bore witness to my plan, now it’s time to take what you’ve learned here today, go forth, and be the villain!

• I know this wasn’t exactly the most fun talk, but I think it’s an important one, and hopefully you did to
• Thank you for your time and attention, and
• Thank you also to the captioning and ASL interpretation services, and to Deque for allowing me to have this great opportunity

• The foundation for this talk was sourced from an article I wrote a couple of years ago for 24ways
• I’ll have a link to it, as well as links to the other resources I used to make this talk included on the slides that are now live on Notist
• I’ll also be linking to this presentation from both my personal website, ericwbailey.design
• And from my Twitter account, @ ericwbailey, if you do the Twitter thing
• And with that, I’ll happily try to answer any questions you have, thank you!