Sensory Friendly Monitoring: Keeping the Noise Down

A presentation at Hops ‘n Ops - Hosted by ChaosSearch, Logz.io, and ThreatStack in March 2019 in Boston, MA, USA by Quintessence Anx

Slide 1

Slide 1

Sensory Friendly Monitoring Keeping the Noise Down QuintessenceAnx

Slide 2

Slide 2

When we try to know everything… QuintessenceAnx // logzio

Slide 3

Slide 3

Too much noise can… …bury important / high severity alerts in a sea of low priority notices …causing engineering teams to start muting alarms or whole alarm sources …which in turn means the people who need to be notified, won’t be. QuintessenceAnx // logzio

Slide 4

Slide 4

Meanwhile, when we turn the dial too far… QuintessenceAnx // logzio

Slide 5

Slide 5

Let’s find a happy medium QuintessenceAnx // logzio All alerts are fictional.

Slide 6

Slide 6

Consider: the cost of noise QuintessenceAnx // logzio

Slide 7

Slide 7

Your Brain on Alerts QuintessenceAnx // logzio Base image credit: Dreamstime

Slide 8

Slide 8

Time Cost ~25 minutes QuintessenceAnx // logzio

Slide 9

Slide 9

Quality Cost QuintessenceAnx // logzio Source: Mo Selim Art Mo Selim Art Speed Challenge

Slide 10

Slide 10

Cost of Multitasking QuintessenceAnx // logzio Image credit: pngtree

Slide 11

Slide 11

So how do we reduce the noise? QuintessenceAnx // logzio

Slide 12

Slide 12

Be Aware, Not Overwhelmed Determine the sources of noise Categorize the types of noise Channel the noise into a productive workflow Create a routine to clear the clutter QuintessenceAnx // logzio

Slide 13

Slide 13

Sources of Noise Logging / alert system Knowledge base Ticketing system Chat integrations Repetition …and you QuintessenceAnx // logzio

Slide 14

Slide 14

Wait, I need to be aware of myself? Absolutely. QuintessenceAnx // logzio All alerts are fictional.

Slide 15

Slide 15

How often do you… …check your email? …check your social media? …check your text messages? …check your Apple / Google messages? … the list goes on. QuintessenceAnx // logzio All alerts are fictional.

Slide 16

Slide 16

Communication & Boundaries Plan for set times to focus on your work and mute non-critical alerts This includes messages from friends & family When setting boundaries make sure your friends, family, and coworkers know what you consider to be relevant emergencies Set reasonable expectations for yourself and others QuintessenceAnx // logzio

Slide 17

Slide 17

But what about external sources of noise? QuintessenceAnx // logzio All alerts are fictional.

Slide 18

Slide 18

Start Categorizing Your Noise False positives False negatives Fragility Frequency (just fix it) QuintessenceAnx // logzio

Slide 19

Slide 19

Save Time: Create Your Noise Flow What needs to be known Who needs to know it How soon should they know How should they be notified QuintessenceAnx // logzio

Slide 20

Slide 20

Re-Evaluate Redundancy Know How to Add a Little Complexity to Stop a Vacuum a.k.a. A bad day in SlackOps (Sorry Slack.) QuintessenceAnx // logzio

Slide 21

Slide 21

Resilient Noise Builds Trust How reliable are your tools and services? How much notification duplication is needed? Do you have the ability to switch alert endpoints in the event of a service outage? Do you regularly evaluate the reliability of your services (external and internal)? QuintessenceAnx // logzio All alerts are fictional.

Slide 22

Slide 22

Keep Alerts Relevant: Sprint Cleaning For every alert triggered, ask: Was the notification needed? How was the incident resolved? Can the solution be automated? Is the solution permanent? How urgently was a solution needed? QuintessenceAnx // logzio Photo by James Pond on Unsplash

Slide 23

Slide 23

Additional Reading “The Cost of Interrupted Work: More Speed and Stress” — Gloria Mark, dept of Informatics @ UC Irvine https://www.ics.uci.edu/~gmark/chi08-mark.pdf “Are digital distractions harming labour productivity?” — The Economist https://www.economist.com/finance-and-economics/2017/12/07/are-digital-distractions-harming-labourproductivity “Brief Interruptions Spawn Errors” — Michigan State University https://msutoday.msu.edu/news/2013/brief-interruptions-spawn-errors/ “Tenets of SRE” — Stephen Thorne, Sr Google SRE https://medium.com/@jerub/tenets-of-sre-8af6238ae8a8 QuintessenceAnx // logzio

Slide 24

Slide 24

QuintessenceAnx Developer Advocate ! QuintessenceAnx // logzio Photo by Hanny Naibaho on Unsplash