Human-in-the-Loop DevOps

A presentation at Failover Conf by Taylor Barnett

Taylor Barnett
@tbarn

1 / 30

Within DevOps, automation has become a North Star. We want to automate the toil away, but the goal of “no toil” is unattainable. Human-in-the-Loop DevOps is the idea that we can benefit from automation while still embracing the human interaction in specific tasks.

Video

Resources

The following resources were mentioned during the presentation or are useful additional information.

Paper: Ironies of Automation by Lisanne Bainbridge
Do-nothing Scripts

Example: A form of an interactive runbook
Paper: How Complex Systems Fail by Richard I. Cook, MD
Article: Above the Line, Below the Line by Richard I. Cook, M.D.
Blog post: 20 Years Since Y2K: The Evolution of IT Operations and DevOps

A blog post I wrote about earlier this year on the growing complexity of the systems we work in
Transposit

Where I work

Buzz and feedback

Here’s what was said about this presentation on social media.

Very inspiring talk by @taylor_atx on how to embrace human factor in automation, improve oncall experience and reduce cognitive load. I loved all Teams & Culture theme talks. We are all tech pros, but we seriously need more of this! #FailoverConf @GremlinInc
— ブラミ (@blami) April 21, 2020
We're loving all of this #DevOps content at #FailoverConf cc: @taylor_atx pic.twitter.com/J2cgn9GJqC
— Gremlin (@GremlinInc) April 21, 2020
Great talk on Human in the loops DevOps by @taylor_atx at #failoverConf hosted by @GremlinInc. Truth right there 👇 pic.twitter.com/QiUqerF8Bn
— Eric Hunsberger (@HunsbergerEric) April 21, 2020
I really enjoyed @taylor_atx's presentation on "Human-in-the-Loops DevOps" during today's #FailoverConf! @GremlinInc
— Jay R. Newlin (@JayRNewlin) April 21, 2020
Reducing cognitive load in incident management - really important for “human-in-the-loop” ops. Music to my soul on how we make SRE and Ops tasks easier @taylor_atx #FailoverConf
— Doug Odegaard (@dodegaard) April 21, 2020
Y’all should read Ironies of Automation if you haven’t. @taylor_atx #FailoverConf pic.twitter.com/snPMuT4ndJ
— Rich Burroughs (@richburroughs) April 21, 2020
Computers can’t see the larger sociotechnical systems. When we look at the system with humans included, it looks different. @taylor_atx #FailoverConf
— Rich Burroughs (@richburroughs) April 21, 2020
#FailoverConf @taylor_atx @trans: Reduce cognitive load: surface contexts, reduce coordination difficulties, work to serve the humans in your loop.
— Heidi, The Sticker Thoughtleader (@wiredferret) April 21, 2020
#failoverconf Reducing cognitive load, put less pressure on people. During incidents, it can feel like you're playing the "telephone game", hard to get shared understanding & coordinate. Use human-in-the-loop to surface useful info, make the on call experience better @taylor_atx
— lisacrispin (@lisacrispin) April 21, 2020
#FailoverConf @taylor_atx @trans: You don't want to be doing new work, you want to be working through the things that you already have. This is not a "new tool" problem until you understand what you're doing now.
— Heidi, The Sticker Thoughtleader (@wiredferret) April 21, 2020
These thoughts about humans and how they participate in automation are very interesting. #FailoverConf
— Rich Burroughs (@richburroughs) April 21, 2020
I appreciate the idea of keeping the human-in-the-loop at critical junctures for maximal value while automating the tedious in the talk by @taylor_atx at #FailoverConf by @GremlinInc. Save human intelligence for use where it is most beneficial while valuing automation elsewhere.
— Yes, that Matthew Helmke (@matthewhelmke) April 21, 2020
#FailoverConf @taylor_atx @trans: We're trying to automate away all the toil, but just because something can be automated doesn't mean it should be. What is the risk of automating something, or leaving it manual?
— Heidi, The Sticker Thoughtleader (@wiredferret) April 21, 2020
#failoverconf Use existing tooling and checklists, add interactive steps to checklists - eg. have scripts in a repo as part of it. Scripts to guide you, they can be versioned and updated like code. @taylor_atx Automate the toil where appropriate (I love her hand drawn slides!)
— lisacrispin (@lisacrispin) April 21, 2020
#FailoverConf @taylor_atx @trans: Telling a developer that a build has failed is something we already know how to do, but what if we surface the things that help identify the problem, like the logs, known points of failure, contextual data?
— Heidi, The Sticker Thoughtleader (@wiredferret) April 21, 2020
#FailoverConf @taylor_atx @trans: When we have an incident, we need to transform an experience into a form of expertise that can be used by others.
— Heidi, The Sticker Thoughtleader (@wiredferret) April 21, 2020
#failoverconf Debería ser obligatorio leerse este documento: https://t.co/GQKuiEHPsq Entenderíamos muchos de los problemas que deberemos afrontar en nuestros equipos. pic.twitter.com/nCkG7dA0Va
— vthot4 (@vthot4) April 21, 2020
#failoverconf In the middle of an incident (and the night), it's hard to understand what automation is doing, what decisions it is making, the context of what's going on when you're paged. Interactive runbooks help the human operator see what controls are available. @taylor_atx
— lisacrispin (@lisacrispin) April 21, 2020
Awesome talk from @taylor_atx, and this slide is a great illustration!#FailoverConf pic.twitter.com/o8BVpDGfyg
— Matt "Garbage File" Stratton (@mattstratton) April 21, 2020
#failoverconf Human-in-the-loop DevOps with @taylor_atx Humans are definitely a theme today - so happy to see this! pic.twitter.com/GL4o7ab2aX
— lisacrispin (@lisacrispin) April 21, 2020
#FailoverConf @GremlinInc
Amazing conference🌱 - entirely online on a subject very close to my heart! Greetings from Germany. #StayAtHome pic.twitter.com/QV3BSm1JTe
— Kamil (@kamil_x451) April 21, 2020
per @taylor_atx “automation” is not one and done, it’s a spectrum and journey. From human in the loop to ML and #AIOps - #sre #FailoverConf
— Chris Riley ☁ (@HoardingInfo) April 21, 2020
if you haven't heard of @transposit now is a great time to become familiar. @taylor_atx will be discussing the ironies of automation and other fun topics at #failoverconf https://t.co/vkoRfvhufS pic.twitter.com/mZA7MbtVSo
— Gremlin (@GremlinInc) April 21, 2020

Human-in-the-Loop DevOps

Link for this presentation:

HTML code for embedding:

Share on social media:

Video

Resources

Paper: Ironies of Automation by Lisanne Bainbridge

Do-nothing Scripts

Paper: How Complex Systems Fail by Richard I. Cook, MD

Article: Above the Line, Below the Line by Richard I. Cook, M.D.

Blog post: 20 Years Since Y2K: The Evolution of IT Operations and DevOps

Transposit

Buzz and feedback