Sensory Friendly Monitoring: Keeping the Noise Down

A presentation at DevOps Days Minneapolis in July 2021 in Minneapolis, MN, USA by Quintessence Anx

Slide 1

Slide 1

Sensory Friendly Monitoring Keep the Noise Down Quintessence Anx Developer Advocate

Slide 2

Slide 2

QuintessenceAnx

Slide 3

Slide 3

Too much noise can • Bury important / high SEV alerts in a sea of low priority notifications • …causing teams to mute individual notifications or their sources. • …meaning that the people who needed to be notified, won’t be. QuintessenceAnx

Slide 4

Slide 4

Too much noise can • Bury important / high SEV alerts in a sea of low priority notifications • …causing teams to mute individual notifications or their sources. • …meaning that the people who needed to be notified, won’t be. QuintessenceAnx

Slide 5

Slide 5

QuintessenceAnx

Slide 6

Slide 6

Reduce the noise by being aware • Determine the sources of noise • Categorize the types of noise • Channel the noise into a productive workflow • Create a routine to clear the clutter QuintessenceAnx

Slide 7

Slide 7

Your brain on alerts QuintessenceAnx

Slide 8

Slide 8

Cost: time and quality degradation QuintessenceAnx

Slide 9

Slide 9

Our brains are not meant to multitask QuintessenceAnx

Slide 10

Slide 10

Sources of noise QuintessenceAnx

Slide 11

Slide 11

How often do you… • …check your email? • …check your social media? • …check your text messages? • …check your Apple/Google messages? • …what about Google Home/Amazon Echo/etc.? • …the list goes on. All alerts are fictional. QuintessenceAnx

Slide 12

Slide 12

Communication & Boundaries • Set times to focus on work and mute non-critical alerts. • This includes messages from friends & family • Define “relevant emergency” • Set reasonable expectations for yourself and others. QuintessenceAnx

Slide 13

Slide 13

Communication & Boundaries • Set times to focus on work and mute non-critical alerts. • This includes messages from friends & family • Define “relevant emergency” • Set reasonable expectations for yourself and others. QuintessenceAnx

Slide 14

Slide 14

Categorize External Sources of Noise • False positives • False negatives • Fragility • Frequency QuintessenceAnx

Slide 15

Slide 15

Noise Flow • What needs to be known • Who needs to know it • How soon should they know • How should they be notified QuintessenceAnx

Slide 16

Slide 16

Re-Evaluate Redundancy How to add a little complexity to stop a vacuum a.k.a. A bad day in ChatOps QuintessenceAnx

Slide 17

Slide 17

Reliability & Redundancy • How reliable are your tools and services? • How much notification duplication is needed? • Can you switch alert endpoints in the event of a service outage? • Do you regularly evaluate the reliability of your services (internal and external)? QuintessenceAnx

Slide 18

Slide 18

For every alert triggered, ask • Was the notification needed? • How was the incident resolved? • Can the solution be automated? • Is the solution permanent? • How urgently is the solution needed? QuintessenceAnx

Slide 19

Slide 19

For every alert triggered, ask • Was the notification needed? • How was the incident resolved? • Can the solution be automated? • Is the solution permanent? • How urgently is the solution needed? QuintessenceAnx

Slide 20

Slide 20

Thank you! Quintessence Anx Developer Advocate 🥑 @ https://noti.st/quintessence QuintessenceAnx