A presentation at DevOpsDays New York City 2019 in in New York, NY, USA by Quintessence Anx
Sensory Friendly Monitoring Keeping the Noise Down QuintessenceAnx
When we try to know everything… QuintessenceAnx // logzio
Too much noise can… …bury important / high severity alerts in a sea of low priority notices …causing engineering teams to start muting alarms or whole alarm sources …which in turn means the people who need to be notified, won’t be. QuintessenceAnx // logzio
Meanwhile, when we turn the dial too far… QuintessenceAnx // logzio
Let’s find a happy medium QuintessenceAnx // logzio All alerts are fictional.
Consider: the cost of noise QuintessenceAnx // logzio
Your Brain on Alerts QuintessenceAnx // logzio Base image credit: Dreamstime
Time Cost ~25 minutes QuintessenceAnx // logzio
Quality Cost QuintessenceAnx // logzio Source: Mo Selim Art Mo Selim Art Speed Challenge
Cost of Multitasking QuintessenceAnx // logzio Image credit: pngtree
So how do we reduce the noise? QuintessenceAnx // logzio
Be Aware, Not Overwhelmed Determine the sources of noise Categorize the types of noise Channel the noise into a productive workflow Create a routine to clear the clutter QuintessenceAnx // logzio
Sources of Noise Logging / alert system Knowledge base Ticketing system Chat integrations Repetition …and you QuintessenceAnx // logzio
Wait, I need to be aware of myself? Absolutely. QuintessenceAnx // logzio All alerts are fictional.
How often do you… …check your email? …check your social media? …check your text messages? …check your Apple / Google messages? … the list goes on. QuintessenceAnx // logzio All alerts are fictional.
Communication & Boundaries Plan for set times to focus on your work and mute non-critical alerts This includes messages from friends & family When setting boundaries make sure your friends, family, and coworkers know what you consider to be relevant emergencies Set reasonable expectations for yourself and others QuintessenceAnx // logzio
But what about external sources of noise? QuintessenceAnx // logzio All alerts are fictional.
Start Categorizing Your Noise False positives False negatives Fragility Frequency (just fix it) QuintessenceAnx // logzio
Save Time: Create Your Noise Flow What needs to be known Who needs to know it How soon should they know How should they be notified QuintessenceAnx // logzio
Re-Evaluate Redundancy Know How to Add a Little Complexity to Stop a Vacuum a.k.a. A bad day in SlackOps (Sorry Slack.) QuintessenceAnx // logzio
Resilient Noise Builds Trust How reliable are your tools and services? How much notification duplication is needed? Do you have the ability to switch alert endpoints in the event of a service outage? Do you regularly evaluate the reliability of your services (external and internal)? QuintessenceAnx // logzio All alerts are fictional.
Keep Alerts Relevant: Sprint Cleaning For every alert triggered, ask: Was the notification needed? How was the incident resolved? Can the solution be automated? Is the solution permanent? How urgently was a solution needed? QuintessenceAnx // logzio Photo by James Pond on Unsplash
Additional Reading “The Cost of Interrupted Work: More Speed and Stress” — Gloria Mark, dept of Informatics @ UC Irvine https://www.ics.uci.edu/~gmark/chi08-mark.pdf “Are digital distractions harming labour productivity?” — The Economist https://www.economist.com/finance-and-economics/2017/12/07/are-digital-distractions-harming-labourproductivity “Brief Interruptions Spawn Errors” — Michigan State University https://msutoday.msu.edu/news/2013/brief-interruptions-spawn-errors/ “Tenets of SRE” — Stephen Thorne, Sr Google SRE https://medium.com/@jerub/tenets-of-sre-8af6238ae8a8 QuintessenceAnx // logzio
QuintessenceAnx Developer Advocate ! QuintessenceAnx // logzio Photo by Hanny Naibaho on Unsplash
The ability to monitor infrastructure has been exploding with new tools on the market and new integrations, so the tools can speak to one another, leading to even more tools, and to a hypothetically very loud monitoring environment with various members of the engineering team finding themselves muting channels, individual alerts, or even alert sources so they can focus long enough to complete other tasks. There has to be a better way - a way to configure comprehensive alerts that send out notifications with the appropriate level of urgency to the appropriate persons at the appropriate time. And in fact there is: during this talk I’ll be walking through different alert patterns and discussing: what we need to know, who needs to know it, as well as how soon and how often do they need to know.
Here’s what was said about this presentation on social media.
for free. You
can too.