A presentation at DevOpsDays Detroit in in Detroit, MI, USA by Quintessence Anx
SENSORY FRIENDLY MONITORING Keeping the noise down QuintessenceAnx
@QuintessenceAnx /@AppDynamics When we try to know everything…
@QuintessenceAnx /@AppDynamics Too much noise can… …bury important / high severity alerts in a sea of low priority notices …causing engineering teams to start muting alarms or whole alarm sources …which in turn means the people who need to be notified, won’t be.
@QuintessenceAnx /@AppDynamics Meanwhile, when we turn the dial too far…
@QuintessenceAnx /@AppDynamics Let’s find a happy medium. All alerts are fictional.
@QuintessenceAnx /@AppDynamics Consider: the cost of noise
@QuintessenceAnx /@AppDynamics Your brain on alerts Base image credit: Dreamstime
@QuintessenceAnx /@AppDynamics Time cost? ~25 minutes
@QuintessenceAnx /@AppDynamics Quality cost Source: Mo Selim Art Mo Selim Art Speed Challenge
@QuintessenceAnx /@AppDynamics Cost of multitasking Image credit: pngtree
@QuintessenceAnx /@AppDynamics So how to reduce the noise?
@QuintessenceAnx /@AppDynamics Be aware, not overwhelmed Determine the sources of noise Categorize the types of noise Channel the noise into a productive workflow Create a routine to clear the clutter
@QuintessenceAnx /@AppDynamics Sources of noise
@QuintessenceAnx /@AppDynamics Wait, I need to be aware of myself? Absolutely. All alerts are fictional.
@QuintessenceAnx /@AppDynamics How often do you… …check your email? …check your social media? …check your text messages? …check your Apple / Google messages? … the list goes on. All alerts are fictional.
@QuintessenceAnx /@AppDynamics Communication & Boundaries Plan for set times to focus on your work and mute non-critical alerts This includes messages from friends & family When setting boundaries make sure your friends, family, and co-workers know what you consider to be relevant emergencies Set reasonable expectations for yourself and others
@QuintessenceAnx /@AppDynamics But what about external sources of noise? All alerts are fictional.
@QuintessenceAnx /@AppDynamics Start categorizing your noise False positives False negatives Fragility Frequency (just fix it)
@QuintessenceAnx /@AppDynamics Save time: create your noise flow What needs to be known Who needs to know it How soon should they know How should they be notified
@QuintessenceAnx /@AppDynamics Re-Evaluate Redundancy Know How to Add a Little Complexity to Stop a Vacuum a.k.a. A bad day in ChatOps
@QuintessenceAnx /@AppDynamics Resilient noise builds trust How reliable are your tools and services? How much notification duplication is needed? Do you have the ability to switch alert endpoints in the event of a service outage? Do you regularly evaluate the reliability of your services (external and internal)? All alerts are fictional.
@QuintessenceAnx /@AppDynamics Keep alerts relevant: Sprint Cleaning For every alert triggered, ask: Was the notification needed? How was the incident resolved? Can the solution be automated? Is the solution permanent? How urgently was a solution needed? Photo by James Pond on Unsplash
@QuintessenceAnx /@AppDynamics Slides & Additional Resources Available on Notist https://noti.st/quintessence
Thank you! QuintessenceAnx Technical Evangelist ! @ https://noti.st/quintessence All alerts are fictional.
The ability to monitor infrastructure has been exploding with new tools on the market and new integrations, so the tools can speak to one another, leading to even more tools, and to a hypothetically very loud monitoring environment with various members of the engineering team finding themselves muting channels, individual alerts, or even alert sources so they can focus long enough to complete other tasks. There has to be a better way - a way to configure comprehensive alerts that send out notifications with the appropriate level of urgency to the appropriate persons at the appropriate time. And in fact there is: during this talk I’ll be walking through different alert patterns and discussing: what we need to know, who needs to know it, as well as how soon and how often do they need to know.
The following resources were mentioned during the presentation or are useful additional information.
Muzzle App is written for macOS to automagically put your computer in DND mode whenever you open a video / sharing app such as Zoom, WebEx, etc.
… they also have a very enjoyable scrolling set of notifications on their webpage.
for free. You
can too.