Sensory Friendly Monitoring: Keeping the Noise Down

A presentation at DevOpsDays Toronto in May 2019 in Toronto, ON, Canada by Quintessence Anx

Slide 1

Slide 1

SENSORY FRIENDLY MONITORING Keeping the noise down QuintessenceAnx

Slide 2

Slide 2

@QuintessenceAnx /@logzio When we try to know everything…

Slide 3

Slide 3

@QuintessenceAnx /@logzio Too much noise can… …bury important / high severity alerts in a sea of low priority notices …causing engineering teams to start muting alarms or whole alarm sources …which in turn means the people who need to be notified, won’t be.

Slide 4

Slide 4

@QuintessenceAnx /@logzio Meanwhile, when we turn the dial too far…

Slide 5

Slide 5

@QuintessenceAnx /@logzio Let’s find a happy medium. All alerts are fictional.

Slide 6

Slide 6

@QuintessenceAnx /@logzio Consider: the cost of noise

Slide 7

Slide 7

@QuintessenceAnx /@logzio Your brain on alerts Base image credit: Dreamstime

Slide 8

Slide 8

@QuintessenceAnx /@logzio Time cost? ~25 minutes

Slide 9

Slide 9

@QuintessenceAnx /@logzio Quality cost Source: Mo Selim Art Mo Selim Art Speed Challenge

Slide 10

Slide 10

@QuintessenceAnx /@logzio Cost of multitasking Image credit: pngtree

Slide 11

Slide 11

@QuintessenceAnx /@logzio So how to reduce the noise?

Slide 12

Slide 12

@QuintessenceAnx /@logzio Be aware, not overwhelmed Determine the sources of noise Categorize the types of noise Channel the noise into a productive workflow Create a routine to clear the clutter

Slide 13

Slide 13

@QuintessenceAnx /@logzio Sources of noise

Slide 14

Slide 14

@QuintessenceAnx /@logzio Wait, I need to be aware of myself? Absolutely. All alerts are fictional.

Slide 15

Slide 15

@QuintessenceAnx /@logzio How often do you… …check your email? …check your social media? …check your text messages? …check your Apple / Google messages? … the list goes on. All alerts are fictional.

Slide 16

Slide 16

@QuintessenceAnx /@logzio Communication & Boundaries Plan for set times to focus on your work and mute non-critical alerts This includes messages from friends & family When setting boundaries make sure your friends, family, and co-workers know what you consider to be relevant emergencies Set reasonable expectations for yourself and others

Slide 17

Slide 17

@QuintessenceAnx /@logzio But what about external sources of noise? All alerts are fictional.

Slide 18

Slide 18

@QuintessenceAnx /@logzio Start categorizing your noise False positives False negatives Fragility Frequency (just fix it)

Slide 19

Slide 19

@QuintessenceAnx /@logzio Save time: create your noise flow What needs to be known Who needs to know it How soon should they know How should they be notified

Slide 20

Slide 20

@QuintessenceAnx /@logzio Re-Evaluate Redundancy Know How to Add a Little Complexity to Stop a Vacuum a.k.a. A bad day in SlackOps (Sorry Slack.)

Slide 21

Slide 21

@QuintessenceAnx /@logzio Resilient noise builds trust How reliable are your tools and services? How much notification duplication is needed? Do you have the ability to switch alert endpoints in the event of a service outage? Do you regularly evaluate the reliability of your services (external and internal)? All alerts are fictional.

Slide 22

Slide 22

@QuintessenceAnx /@logzio Keep alerts relevant: Sprint Cleaning For every alert triggered, ask: Was the notification needed? How was the incident resolved? Can the solution be automated? Is the solution permanent? How urgently was a solution needed? Photo by James Pond on Unsplash

Slide 23

Slide 23

Additional Reading @QuintessenceAnx /@logzio “The Cost of Interrupted Work: More Speed and Stress” — Gloria Mark, dept of Informatics @ UC Irvine https://www.ics.uci.edu/~gmark/chi08-mark.pdf “Are digital distractions harming labour productivity?” — The Economist https://www.economist.com/finance-and-economics/2017/12/07/ are-digital-distractions-harming-labour-productivity “Brief Interruptions Spawn Errors” — Michigan State University https://msutoday.msu.edu/news/2013/brief-interruptions-spawn-errors/ “Tenets of SRE” — Stephen Thorne, Sr Google SRE https://medium.com/@jerub/tenets-of-sre-8af6238ae8a8

Slide 24

Slide 24

@QuintessenceAnx /@logzio Slides Available on Notist https://noti.st/quintessence

Slide 25

Slide 25

Thank you! QuintessenceAnx US Developer Advocate 🥑 All alerts are fictional.