A presentation at Conf42: Site Reliability Engineering 2022 by Richard
LMAO Helps During Outages
Richard Lewis www.gogorichie.com • Sr. DevOps Consultant • 20+ years of working with Operations and Software Development Team • Organizer of Chicago Monitoring Enthusiast meetup community • Diehard Chicago White Sox fan!
3Cloud is the largest pureplay Azure services partner in the world. UNRIVALED IMPACT No other Microsoft partner invest more resources to create the ultimate Azure experience. $24M $117M $38M FY22 YTD ACR Influenced FY22 YTD FY22 YTD ADS Influenced APP Influence
100% 650+ 600 Annual Growth over 5 years FY22 YTD Co-sell Wins full-time, U.S. based Azure experts 600+ 15+ Full time U.S. based Azure architects average years consulting for each technical staff member 100% AZURE FOCUSED Our expertise in Data & Analytics, App Innovation, and Cloud Platform position us as the most comprehensive pure play Azure partner in the world. PROVEN EXPERTISE 750+ Azure projects delivered to 350+ unique U.S. managed clients in 2021 alone!
Why LMAO?
Things Needed For A Strategy Logs Metrics Alerts Observability Tool
What Are We Talking About? Platform Support Strategies Creating a standard for knowledge sharing Reduction in mean time to resolution (MTTR) Building Psychological confidence within your team
Logs & Metrics • Provide insight into the what and when • What happened? • How many errors • How many request • When did it happen? • How long was the duration
Alerts Pages Tickets Or
Alert Trauma Little Richard On-Call 365 16hrs a day No way to remote connect to office No logging framework Burned out within 6 months
Managing Alerts Effectively Scheduling team members appropriately. Avoid alert fatigue where possible. Collect data on the alerts and look for ways to reduce them regularly. Be cautious when introducing new alerts
Observability • Keep track of KPI’s and SLA’s • Monitor usage • Helps with spotting trends
Preparing The Team Have a playbook Practice for outages
What Should Be In It? • Playbook should located where the on-call team will be able to access it quickly. • Links to application relate observability tools • Details about the golden signals for application. • Relevant information from previously outage. • Application owner contact details • Anything else you might find important.
Preparation and Training
Practice Chaos Engineering Concept created by Netflix. The goal Increase resiliency Identify and address single points of failure early. Gives you the ability to test your documentation and processes. https://techhq.com/2019/03/how-netflix-pioneered-chaos-engineering/
Postmortems
Recap of the Outage • Should be done within a day or so post incident • What went right? • What went wrong? • Where do we get lucky? https://sre.google/sre-book/postmortem-culture/
Take Aways • Have a LMAO strategy in place. • Have the appropriate documentation ready to go. • Update documentation regularly. • Avoid alert fatigue. • Run readiness preparation drills.
Thank You Richard Lewis Twitter: @gogorichie Email: Richard@gogorichie.com
This short fast pace talk is aimed toward’s those who might be on pager duty. Richard will cover the four things that will help you survive any outage nightmare. Cause when your company has an outage no need to sweat just remember to LMAO and you’ll get through it.
The following resources were mentioned during the presentation or are useful additional information.
for free. You
can too.