Sensory Friendly Monitoring: Keeping the Noise Down (Ignite Format)

Sensory Friendly Monitoring Keeping the noise down QuintessenceAnx

When we try to know everything … @QuintessenceAnx // @logzio

Too much noise can … ● … bury important / high severity alerts in a sea of low priority notices ● … cause engineering teams to start muting alarms or whole notification sources ○ … which in turn means the people who need to be notified, aren’t. @QuintessenceAnx // @logzio

Turning the dial down too far… …yields mistrust in the quiet, we are hypervigilant So we need to find a happy medium. @QuintessenceAnx // @logzio

Consider: the cost of noise @QuintessenceAnx // @logzio

Your brain on alerts: osmotic retention @QuintessenceAnx // @logzio

Time Cost ~25 minutes (All alerts are fictional.) @QuintessenceAnx // @logzio

Quality cost @QuintessenceAnx // @logzio

Multitasking Cost @QuintessenceAnx // @logzio

How do we reduce the noise? @QuintessenceAnx // @logzio

Know the sources of noise @QuintessenceAnx // @logzio

Be aware of how you contribute to your noise @QuintessenceAnx // @logzio

Self induced noise Communicate and set boundaries for what works for you (All alerts are fictional.) @QuintessenceAnx // @logzio

External noise Categorize and create workflows around the noise @QuintessenceAnx // @logzio

Categorize and flow ● False negatives ● False positives ● Fragility ● Frequency (fix it) ● What needs to be known ● Who needs to know it ● How soon should they know ● How should they be notified @QuintessenceAnx // @logzio

Resiliency builds trust ● How reliable are your tools and services? ● How much notification duplication is needed? ● Do you have the ability to switch alert endpoints / mechanisms in the event of a service outage? ● Do you regularly evaluate the reliability of your services? @QuintessenceAnx // @logzio

Keep Alerts Relevant: Sprint Cleaning @QuintessenceAnx // @logzio

Presentation available on Github https://github.com/quintessence/presentations @QuintessenceAnx // @logzio

Additional Reading ● “The Cost of Interrupted Work: More Speed and Stress” — Gloria Mark, dept of Informatics @ UC Irvine https://www.ics.uci.edu/~gmark/chi08-mark.pdf ● “Are digital distractions harming labour productivity?” — The Economist https://www.economist.com/finance-and-economics/2017/12/07/are-digital-distractions-harming-labour-productivity ● “Brief Interruptions Spawn Errors” — Michigan State University https://msutoday.msu.edu/news/2013/brief-interruptions-spawn-errors/ ● “Tenets of SRE” — Stephen Thorne, Sr Google SRE https://medium.com/@jerub/tenets-of-sre-8af6238ae8a8 @QuintessenceAnx // @logzio

@QuintessenceAnx Developer Advocate