SENSORY FRIENDLY MONITORING Keeping the noise down QuintessenceAnx
A presentation at CloudOps Summit in July 2020 in by Quintessence Anx
SENSORY FRIENDLY MONITORING Keeping the noise down QuintessenceAnx
@QuintessenceAnx /@PagerDuty When we try to know everything…
@QuintessenceAnx /@PagerDuty Too much noise can… …bury important / high severity alerts in a sea of low priority notices …causing engineering teams to start muting alarms or whole alarm sources …which in turn means the people who need to be notified, won’t be.
@QuintessenceAnx /@PagerDuty Meanwhile, when we turn the dial too far…
@QuintessenceAnx /@PagerDuty Let’s find a happy medium. All alerts are fictional.
@QuintessenceAnx /@PagerDuty Consider: the cost of noise
@QuintessenceAnx /@PagerDuty Your brain on alerts Base image credit: Dreamstime
@QuintessenceAnx /@PagerDuty Time cost? ~25 minutes
@QuintessenceAnx /@PagerDuty Quality cost Source: Mo Selim Art Mo Selim Art Speed Challenge
@QuintessenceAnx /@PagerDuty Cost of multitasking Image credit: pngtree
@QuintessenceAnx /@PagerDuty So how to reduce the noise?
@QuintessenceAnx /@PagerDuty Be aware, not overwhelmed Determine the sources of noise Categorize the types of noise Channel the noise into a productive workflow Create a routine to clear the clutter
@QuintessenceAnx /@PagerDuty Sources of noise
@QuintessenceAnx /@PagerDuty Wait, I need to be aware of myself? Absolutely. All alerts are fictional.
@QuintessenceAnx /@PagerDuty How often do you… …check your email? …check your social media? …check your text messages? …check your Apple / Google messages? … the list goes on. All alerts are fictional.
@QuintessenceAnx /@PagerDuty Communication & Boundaries Plan for set times to focus on your work and mute non-critical alerts This includes messages from friends & family When setting boundaries make sure your friends, family, and co-workers know what you consider to be relevant emergencies Set reasonable expectations for yourself and others
@QuintessenceAnx /@PagerDuty But what about external sources of noise? All alerts are fictional.
@QuintessenceAnx /@PagerDuty Start categorizing your noise False positives False negatives Fragility Frequency (just fix it)
@QuintessenceAnx /@PagerDuty Save time: create your noise flow What needs to be known Who needs to know it How soon should they know How should they be notified
@QuintessenceAnx /@PagerDuty Re-Evaluate Redundancy Know How to Add a Little Complexity to Stop a Vacuum a.k.a. A bad day in ChatOps
@QuintessenceAnx /@PagerDuty Resilient noise builds trust How reliable are your tools and services? How much notification duplication is needed? Do you have the ability to switch alert endpoints in the event of a service outage? Do you regularly evaluate the reliability of your services (external and internal)? All alerts are fictional.
@QuintessenceAnx /@PagerDuty Keep alerts relevant: Sprint Cleaning For every alert triggered, ask: Was the notification needed? How was the incident resolved? Can the solution be automated? Is the solution permanent? How urgently was a solution needed? Photo by James Pond on Unsplash
@QuintessenceAnx /@PagerDuty Slides & Additional Resources Available on Notist https://noti.st/quintessence
Thank you! QuintessenceAnx Developer Advocate 🥑 @ https://noti.st/quintessence All alerts are fictional.
FIN.
SENSORY FRIENDLY MONITORING Keeping the noise down QuintessenceAnx
@QuintessenceAnx /@PagerDuty Re-Evaluate Redundancy Know How to Add a Little Complexity to Stop a Vacuum a.k.a. A bad day in SlackOps (Sorry Slack.)
Additional Reading @QuintessenceAnx /@PagerDuty “The Cost of Interrupted Work: More Speed and Stress” — Gloria Mark, dept of Informatics @ UC Irvine https://www.ics.uci.edu/~gmark/chi08-mark.pdf “Are digital distractions harming labour productivity?” — The Economist https://www.economist.com/finance-and-economics/2017/12/07/ are-digital-distractions-harming-labour-productivity “Brief Interruptions Spawn Errors” — Michigan State University https://msutoday.msu.edu/news/2013/brief-interruptions-spawn-errors/ “Tenets of SRE” — Stephen Thorne, Sr Google SRE https://medium.com/@jerub/tenets-of-sre-8af6238ae8a8
