A presentation at Hops ‘n Ops - Hosted by ChaosSearch, Logz.io, and ThreatStack in in Boston, MA, USA by Quintessence Anx
Sensory Friendly Monitoring Keeping the Noise Down QuintessenceAnx
When we try to know everything… QuintessenceAnx // logzio
Too much noise can… …bury important / high severity alerts in a sea of low priority notices …causing engineering teams to start muting alarms or whole alarm sources …which in turn means the people who need to be notified, won’t be. QuintessenceAnx // logzio
Meanwhile, when we turn the dial too far… QuintessenceAnx // logzio
Let’s find a happy medium QuintessenceAnx // logzio All alerts are fictional.
Consider: the cost of noise QuintessenceAnx // logzio
Your Brain on Alerts QuintessenceAnx // logzio Base image credit: Dreamstime
Time Cost ~25 minutes QuintessenceAnx // logzio
Quality Cost QuintessenceAnx // logzio Source: Mo Selim Art Mo Selim Art Speed Challenge
Cost of Multitasking QuintessenceAnx // logzio Image credit: pngtree
So how do we reduce the noise? QuintessenceAnx // logzio
Be Aware, Not Overwhelmed Determine the sources of noise Categorize the types of noise Channel the noise into a productive workflow Create a routine to clear the clutter QuintessenceAnx // logzio
Sources of Noise Logging / alert system Knowledge base Ticketing system Chat integrations Repetition …and you QuintessenceAnx // logzio
Wait, I need to be aware of myself? Absolutely. QuintessenceAnx // logzio All alerts are fictional.
How often do you… …check your email? …check your social media? …check your text messages? …check your Apple / Google messages? … the list goes on. QuintessenceAnx // logzio All alerts are fictional.
Communication & Boundaries Plan for set times to focus on your work and mute non-critical alerts This includes messages from friends & family When setting boundaries make sure your friends, family, and coworkers know what you consider to be relevant emergencies Set reasonable expectations for yourself and others QuintessenceAnx // logzio
But what about external sources of noise? QuintessenceAnx // logzio All alerts are fictional.
Start Categorizing Your Noise False positives False negatives Fragility Frequency (just fix it) QuintessenceAnx // logzio
Save Time: Create Your Noise Flow What needs to be known Who needs to know it How soon should they know How should they be notified QuintessenceAnx // logzio
Re-Evaluate Redundancy Know How to Add a Little Complexity to Stop a Vacuum a.k.a. A bad day in SlackOps (Sorry Slack.) QuintessenceAnx // logzio
Resilient Noise Builds Trust How reliable are your tools and services? How much notification duplication is needed? Do you have the ability to switch alert endpoints in the event of a service outage? Do you regularly evaluate the reliability of your services (external and internal)? QuintessenceAnx // logzio All alerts are fictional.
Keep Alerts Relevant: Sprint Cleaning For every alert triggered, ask: Was the notification needed? How was the incident resolved? Can the solution be automated? Is the solution permanent? How urgently was a solution needed? QuintessenceAnx // logzio Photo by James Pond on Unsplash
Additional Reading “The Cost of Interrupted Work: More Speed and Stress” — Gloria Mark, dept of Informatics @ UC Irvine https://www.ics.uci.edu/~gmark/chi08-mark.pdf “Are digital distractions harming labour productivity?” — The Economist https://www.economist.com/finance-and-economics/2017/12/07/are-digital-distractions-harming-labourproductivity “Brief Interruptions Spawn Errors” — Michigan State University https://msutoday.msu.edu/news/2013/brief-interruptions-spawn-errors/ “Tenets of SRE” — Stephen Thorne, Sr Google SRE https://medium.com/@jerub/tenets-of-sre-8af6238ae8a8 QuintessenceAnx // logzio
QuintessenceAnx Developer Advocate ! QuintessenceAnx // logzio Photo by Hanny Naibaho on Unsplash
The ability to monitor infrastructure has been exploding with new tools on the market and new integrations, so the tools can speak to one another, leading to even more tools, and to a hypothetically very loud monitoring environment with various members of the engineering team finding themselves muting channels, individual alerts, or even alert sources so they can focus long enough to complete other tasks. There has to be a better way - a way to configure comprehensive alerts that send out notifications with the appropriate level of urgency to the appropriate persons at the appropriate time. And in fact there is: during this talk I’ll be walking through different alert patterns and discussing: what we need to know, who needs to know it, as well as how soon and how often do they need to know.
Here’s what was said about this presentation on social media.
Thanks to everyone who attended our Hops n Ops meetup with @CHAOSSEARCH and @threatstack! We had a great time learning new approaches to solve challenges of scale, security, and business growth over delicious drinks with #BostonTech professionals. See you next time! pic.twitter.com/qXRCZRK1CX
— Logz.io (@logzio) March 14, 2019
Attention beer and DevOps lovers. Swing by Boston Beer Works on Canal St. on March 13th @ 5:30 for a fun night of drinks and good ops talk with local IT, DevOps, and Security professionals with @CHAOSSEARCH@logzio and @threatstackhttps://t.co/xgV9WbHRSs
— CHAOSSEARCH (@CHAOSSEARCH) March 8, 2019
Thanks everyone for joining us at Boston Beer Works last night with @threatstack and @logzio for some fantastic networking and tech talks. pic.twitter.com/i5FANxqGmC
— CHAOSSEARCH (@CHAOSSEARCH) March 14, 2019
Join us and our friends @threatstack and @logzio at Boston Beer Works on Canal St. on Wednesday, March 13th @ 5:30 for a fun night of drinks, apps, and good ops talk with local IT, DevOps, and Security professionals. https://t.co/xgV9WbHRSs
— CHAOSSEARCH (@CHAOSSEARCH) February 6, 2019
Join us Wednesday March 13th at Boston Beer works as we co-host a meetup with @threatstack and @logzio. Food, drinks, and some fantastic technical talks are all lined up. https://t.co/LkIKGZuCw0
— CHAOSSEARCH (@CHAOSSEARCH) February 6, 2019
We are going to be running an awesome meetup on March 13th in Boston with @threatstack and @logzio. Come and watch some great technical talks, while enjoying some food and drinks and network with your peers!
— CHAOSSEARCH (@CHAOSSEARCH) February 11, 2019
Learn more and RSVP here: https://t.co/LkIKGZuCw0
Join us March 13th in Boston with @threatstack and @logzio. Enjoy food, drinks, and great networking with your peers while watching some fantastic technical talks.
— CHAOSSEARCH (@CHAOSSEARCH) February 13, 2019
Learn more and RSVP here: https://t.co/LkIKGZd17q
We are teaming up with @CHAOSSEARCH and @logzio for a fun night of beer and ops talk at Boston Beer Works on Canal St. Reserve your spot today! https://t.co/MHJTNmQJnx
— Threat Stack, Inc (@threatstack) February 15, 2019
Don't miss the upcoming Boston meetup we are running with @logzio and @threatstack at the Boston Beer Works on March 13th.
— CHAOSSEARCH (@CHAOSSEARCH) February 20, 2019
Free food and drinks and some amazing talks lined up! RSVP here:https://t.co/LkIKGZd17q
This upcoming Wednesday (3/13) we are teaming up with friends at @CHAOSSEARCH and @logzio to bring you a night of apps, drinks, and Ops talk. RSVP today—space is limited! https://t.co/MHJTNmQJnx
— Threat Stack, Inc (@threatstack) March 7, 2019
Join us THIS WEDNESDAY for Hops N Ops Meetup at Boston Beer Works Canal St.
— CHAOSSEARCH (@CHAOSSEARCH) March 11, 2019
Join us along with @threatstack and @logzio for a great night of beer, food, and ops talk
RSVP here - space is limited!https://t.co/LkIKGZuCw0
Tomorrow, join us along with @CHAOSSEARCH and @logzio for a great night of beer, food, and ops talk at Boston Beer Works. RSVP now—you won't want to miss it! https://t.co/MHJTNmQJnx
— Threat Stack, Inc (@threatstack) March 12, 2019
We're only 3 days away from the Hops N Ops Meetup! Join us along with @CHAOSSEARCH and @logzio for a great night of beer and ops talk at Boston Beer Works: https://t.co/MHJTNmQJnx
— Threat Stack, Inc (@threatstack) March 10, 2019
How does a SaaS security company like @threatstack think about security? If you're in the Boston area on March 13th come join us at our Hops n'Ops meetup! You'll get that inside scoop, along with other talks from @logzio and @CHAOSSEARCH as well 🎉 https://t.co/uvbgloUjtR
— PatCable.tif.exe (@patcable) February 26, 2019
If you still have some energy to burn after @ServerlessBOS, I'm talking at hops+ops tomorrow with @patcable AND @QuintessenceAnx. I promise beer, some OPs, some terrible ways to push data to s3, and an animated gif I made. Also corrupted movie quotes.https://t.co/8D0cQCzesl
— Patrick Flaherty (@PlatformPatrick) March 12, 2019
Hope to see a bunch of "Boston-eers" this Wednesday! https://t.co/Jty1o44aPn
— Quintessence {🥑/} (@QuintessenceAnx) March 12, 2019
Speaking at #devopsdays Charlotte today and tomorrow - after that I will be speaking at a joint event with @CHAOSSEARCH and @threatstack . Looking forward to both! https://t.co/mBKt3NNT4w
— Quintessence {🥑/} (@QuintessenceAnx) February 7, 2019