Sensory Friendly Monitoring: Keeping the Noise Down

A presentation at All Day DevOps in November 2020 in by Quintessence Anx

Slide 1

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G NOVEMBER 12, 2020 Quintessence Anx Sensory Friendly Monitoring: Keep the Noise Down

Slide 2

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Head’s up: Mute your Echo devices in listening range, or use 🎧 QuintessenceAnx

Slide 3

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G When we try to know everything… QuintessenceAnx

Slide 4

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G QuintessenceAnx

Slide 5

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Too much noise can QuintessenceAnx

Slide 6

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Bury important / high SEV alerts in a sea of low priority notifications QuintessenceAnx

Slide 7

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G …causing teams to mute individual notifications or their sources. QuintessenceAnx

Slide 8

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G The result? QuintessenceAnx

Slide 9

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G The people who needed to be notified, won’t be. QuintessenceAnx

Slide 10

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Meanwhile, when we turn the dial back too far… QuintessenceAnx

Slide 11

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G QuintessenceAnx

Slide 12

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Let’s find a happy medium. All alerts are fictional. QuintessenceAnx

Slide 13

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Consider: the cost of noise QuintessenceAnx

Slide 14

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Your brain on alerts QuintessenceAnx

Slide 15

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Time cost: ~25 minutes QuintessenceAnx

Slide 16

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Quality Cost QuintessenceAnx

Slide 17

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Multitasking Cost QuintessenceAnx

Slide 18

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G How to reduce the noise? QuintessenceAnx

Slide 19

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Be aware, not overwhelmed. QuintessenceAnx

Slide 20

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Determine the sources of noise QuintessenceAnx

Slide 21

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Categorize the types of noise QuintessenceAnx

Slide 22

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Channel the noise into a productive flow QuintessenceAnx

Slide 23

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Create a routine to clear the clutter. QuintessenceAnx

Slide 24

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Be aware, not overwhelmed • Determine the sources of noise • Categorize the types of noise • Channel the noise into a productive workflow • Create a routine to clear the clutter QuintessenceAnx

Slide 25

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Sources of noise QuintessenceAnx

Slide 26

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Wait, I need to be aware of myself? All alerts are fictional. QuintessenceAnx

Slide 27

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G How often do you… • …check your email? • …check your social media? • …check your text messages? • …check your Apple/Google messages? • …the list goes on. All alerts are fictional. QuintessenceAnx

Slide 28

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Head’s up: Mute your Echo devices in listening range, or use 🎧 QuintessenceAnx

Slide 29

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Communication & Boundaries QuintessenceAnx

Slide 30

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Set times to focus on work and mute non-critical alerts. QuintessenceAnx

Slide 31

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G This includes messages from friends & family QuintessenceAnx

Slide 32

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Define “relevant emergency” QuintessenceAnx

Slide 33

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Set reasonable expectations for yourself and others. QuintessenceAnx

Slide 34

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Communication & Boundaries • Set times to focus on work and mute non-critical alerts. • This includes messages from friends & family • Define “relevant emergency” • Set reasonable expectations for yourself and others. QuintessenceAnx

Slide 35

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G And what about external sources of noise? QuintessenceAnx

Slide 36

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Categorize your noise QuintessenceAnx

Slide 37

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G False positives QuintessenceAnx

Slide 38

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G False negatives QuintessenceAnx

Slide 39

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Fragility QuintessenceAnx

Slide 40

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Frequency (fix it) QuintessenceAnx

Slide 41

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Noise Categories • False positives • False negatives • Fragility • Frequency QuintessenceAnx

Slide 42

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Save time: create your noise flow QuintessenceAnx

Slide 43

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G What needs to be known QuintessenceAnx

Slide 44

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Who needs to know it QuintessenceAnx

Slide 45

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G How soon should they know QuintessenceAnx

Slide 46

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G How should they be notified QuintessenceAnx

Slide 47

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Noise Flow • What needs to be known • Who needs to know it • How soon should they know • How should they be notified QuintessenceAnx

Slide 48

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Re-Evaluate Redundancy How to add a little complexity to stop a vacuum a.k.a. A bad day in ChatOps QuintessenceAnx

Slide 49

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Resilient noise builds trust QuintessenceAnx

Slide 50

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G How reliable are your tools and services? QuintessenceAnx

Slide 51

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G How much notification duplication is needed? QuintessenceAnx

Slide 52

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Can you switch alert endpoints in the event of a service outage? QuintessenceAnx

Slide 53

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Do you regularly evaluate the reliability of your services (internal and external)? QuintessenceAnx

Slide 54

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Reliability & Redundancy • How reliable are your tools and services? • How much notification duplication is needed? • Can you switch alert endpoints in the event of a service outage? • Do you regularly evaluate the reliability of your services (internal and external)? QuintessenceAnx

Slide 55

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Keep alerts relevant: Sprint Cleaning QuintessenceAnx

Slide 56

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G For every alert triggered, ask: QuintessenceAnx

Slide 57

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Was the notification needed? QuintessenceAnx

Slide 58

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G How was the incident resolved? QuintessenceAnx

Slide 59

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Can the solution be automated? QuintessenceAnx

Slide 60

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Is the solution permanent? QuintessenceAnx

Slide 61

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G How urgently was a solution needed? QuintessenceAnx

Slide 62

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G For every alert triggered, ask • Was the notification needed? • How was the incident resolved? • Can the solution be automated? • Is the solution permanent? • How urgently is the solution needed? QuintessenceAnx

Slide 63

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Slides & Additional Resources on Notist https://noti.st/quintessence QuintessenceAnx

Slide 64

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G Thank you! Quintessence Anx Developer Advocate 🥑 @ https://noti.st/quintessence QuintessenceAnx

Slide 65

T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G THANK YOU TO OUR SPONSORS