A presentation at Conf42: SRE by Quintessence Anx
Don’t Panic! Effective Incident Response Presented by @QuintessenceAnx DevOps Advocate 2021 Proprietary & Confidential
What We’ll Be Learning Today: After completing this training, you will be able to: • Build a foundation for an effective incident response process in your organization • Understand suggested practices needed for successful incident response • Identify practices that limit damage and reduce recovery time and costs
An incident is any unplanned disruption or event that requires immediate attention or action
Replace chaos with calm
Incident Response is an organized approach to addressing and managing an incident
The goal of Incident Response is to handle the situation in a way that limits damage and reduces recovery time and costs
To Accomplish this Goal you Must: ● Mobilize and inform only the right people at the right time ● Use systematic learning and improvement ● Work toward total automation
Based on the Incident Command System, originally developed for California wildfire response.
An incident is an unplanned disruption or event that requires immediate attention or action
A major incident requires a coordinated response between multiple teams
The 4 Commonalities of Major Incidents ● Timing is a surprise; little or no warning ● Time matters; need to respond quickly ● Situation rarely perfectly understood at the start ● Require mobilization and coordination, typically cross-functional
SEV-5 SEV-4 SEV-3 Incident SEV-2 Major Incident SEV-1
Anyone can trigger the Incident Response Process at any time
Rich Adams !ic page 11:12 11:12 Officer URL APP Paging Incident Commanders(s) Arup Chakrabarti has been paged. Paul Rechsteiner has been paged. Renee Lung has been paged. Use !ic responders to see who the team responders are. Incident triggered: https://example.pagerduty.com/incident/PD5I34R !ic page
PEACETIME WARTIME
NORMAL EMERGENCY
OK NOT OK
Decision Paralysis
People Roles & Incident Categorization Proprietary & Confidential
The Four Steps of an Incident TRIAGE MOBILIZE RESOLVE PREVENT
Roles of Incident Response COMMAND Incident Commander Scribe Deputy LIAISONS Internal Liaison Customer Liaison OPERATIONS Subject Matter Expert (SME) Subject Matter Expert (SME) Subject Matter Expert (SME) Subject Matter Expert (SME) Subject Matter Expert (SME)
Setting this up at scale For a department-wide Incident Response process, you will need a few things set up to begin. This includes: ● An on-call schedule for a primary and backup Incident Commander (this role is team agnostic) ● On-call schedules for primary and backup subject matter experts (one primary and one backup for each team) ● Additional on-call rotations for other roles ● A method of paging team members (response mobilization)
Incident Response - typical sequence of events !ic page Incident Commander Subject Matter Expert (SME) Usually first individual to respond, and solve (or escalate) Internal Liaison (SME) Scribe Deputy LIAISONS External Liaison (SME) COMMAND (SME) (SME) OPERATIONS Team Bravo Team Charlie Team Delta Team Echo
How Do The Roles Scale Down? For a small team-based Incident Response process, you will need a few things set up to begin. This includes: ● An on-call schedule for primary and backup subject matter experts ● A method of paging out other team members
Small Team Incident Response Incident Commander Scribe Subject Matter Expert (SME) Subject Matter Expert (SME) Primary On-Call Backup On-Call Subject Matter Expert (SME) Subject Matter Expert (SME) Subject Matter Expert (SME)
Incident Commander: Role and Responsibilities Proprietary & Confidential
Replace chaos with calm
Single source of reference
Gain consensus “Are there any strong objections”
Make a decision
Assign tasks to a specific person
Becomes the highest authority (Yes, even higher than the CEO)
Deep technical knowledge is not required
Handoffs are encouraged
ASK FOR STATUS FOLLOW UP ON TASK COMPLETION DECIDE ACTION GAIN CONSENSUS ASSIGN TASK
SIZE-UP VERIFY STABILIZE UPDATE
Quick Tips for New Incident Commanders ● ● ● ● ● ● Introduce yourself on the call with your name and that you are the Incident Commander Avoid acronyms Speak slowly and with purpose On the call, kick people off if they are being disruptive Time-box tasks and check in for status updates Explicitly declare when the response has ended
Summary: Importance of the Incident Commander • Keeps everyone focused • Keeps decision-making moving • Helps to avoid the bystander effect • Keep things moving towards a resolution during a major incident
Roles of Incident Response COMMAND Incident Commander Scribe Deputy LIAISONS Internal Liaison Customer Liaison OPERATIONS Subject Matter Expert (SME) Subject Matter Expert (SME) Subject Matter Expert (SME) Subject Matter Expert (SME) Subject Matter Expert (SME)
Importance of the Deputy Role • Keeps the Incident Commander focused • Takes on any and all additional tasks as necessary • Serves to follow up on reminders and ensure tasks aren’t missed • Acts as a “hot standby” for the Incident Commander
Importance of the Scribe • Documents the incident timeline and important events as they occur • The incident log will be used during the post-mortem process • Note when important actions are taken, follow-up items, and status updates • Anyone can be a Scribe
Importance of the Communications Liaison Roles • Can be external, internal, or both • Notifies customers of current conditions, and informs the Incident Commander of relevant feedback • Crafts language appropriate status updates and notification messages • Typically a member of the Support team
Incident Response Pitfalls Proprietary & Confidential
Executive Swoop
“Let’s try and resolve this in 10 minutes please!”
“Can I get a spreadsheet of all affected customers?”
“Do what I say”
KEY TAKEAWAY Do you wish to take command? …
Failure to Notify Stakeholders
Too frequent status updates
Red Herrings
Anti-Patterns ● ● ● ● ● Debating the severity of an incident during the call Discussing process and policy decisions Not disseminating policy changes Hesitating to escalate to other responders Neglecting the postmortem and follow up activities ● ● ● ● ● Trying to take on multiple roles Not disseminating policy changes Getting everyone on the call Forcing everyone to stay on the call Assuming silence means no progress
How do I prepare to manage incident response teams?
Step 1 Ensure explicit processes and expectations exist
Step 2 Practice running major incidents as a team
Step 3 Find ways to tune your processes for your teams to work
Step 4 Make Checklists
Example Checklists Start of Incident: Mobilize Response ❏ ❏ ❏ ❏ ❏ ❏ ❏ ❏ Join the #incident-war-room and Zoom call Announce self as Incident Commander Acknowledge the incident Assign deputy Assign scribe Confirm liaison present Confirm SMEs present Run !ic responders to get list of oncalls on Slack Incident Response Loop ❏ ❏ Size-up the situation ❏ What’s wrong? ❏ Which systems are affected? ❏ Is this affecting multiple systems? ❏ What’s the customer impact? Stabilize the incident ❏ What actions can we take? ❏ Was there a related change or deploy? Reminders during an Ongoing Incident ❏ ❏ ❏ Suggest people leave call if they are not required SME, Scribe, Comms handoff to avoid fatigue Incident Commander Swap ❏ Ask deputy to take over ❏ Summarize status ❏ Announce change in command Incident Resolved ❏ ❏ ❏ ❏ ❏ Notify customers of resolution Scale down the response ❏ Direct all follow up to #incident-followup ❏ Announce end of incident call Resolve the PD incident Create the postmortem ❏ Assign postmortem owner Send email to incident-reports@pd.com
Don’t neglect the postmortem
Postmortems for Beginners • A Brief Overview: high level of the impact (1-2 sentences) • What happened: Detailed description, usually 1-2 paragraphs or more depending on length of response efforts • What went well? • What didn’t go so well? • Action items - if you don’t have any, what was the point of having a response?
Detailed Postmortems • Brief Overview: high level of the impact (1-2 sentences) • Contributing factors • Resolution actions • What Happened: Detailed description (usually 1-2 paragraphs, or more) • Impact: who did this affect, by how much, for how long? • What went well • Internal Messaging • What didn’t go so well • External Messaging (direct either to affected customers or all customers) • Action Items (if you don’t have any, what was the point of having a response?) • Detailed Timeline of Events
Summary • Use the Incident Command System for managing incidents • An Incident Commander takes charge during wartime scenarios • Set expectations upward • Work with your team to set explicit processes and expectations • Practice, practice, practice! • Don’t forget to review and improve
response.pagerduty.com Proprietary & Confidential
Q&A @QuintessenceAnx https://noti.st/quintessence
In today’s world, service downtime has a significant impact on any business. Being able to respond quickly and proactively to issues can dramatically reduce the repercussions of any incident, both financial and reputational.
We’ll take lessons learned from formalized incident response, such as those used by first responders, and show you how to apply those same practices to your organization. By utilizing these methods you’ll improve both the speed and effectiveness of your team’s response, reducing the amount of downtime experienced.
In this workshop, attendees will:
The following resources were mentioned during the presentation or are useful additional information.
This documentation covers parts of the PagerDuty Incident Response process. It is a cut-down version of our internal documentation used at PagerDuty for any major incidents and to prepare new employees for on-call responsibilities. It provides information not only on preparing for an incident, but also what to do during and after the incident. It is intended to be used by on-call practitioners and those involved in an operational incident response process (or those wishing to enact a formal incident response process). See the about page for more information on what this documentation is and why it exists.
Full-service ownership means that people take responsibility for supporting the software they deliver, at every stage of the software/service lifecycle. That level of ownership brings development teams much closer to their customers, the business, and the value being delivered. This guide provides a step-by-step process that software delivery operations teams can use to ensure they are able to fully own their services.
The postmortem concept is well known in the technology industry, but it can be difficult for newer individuals, teams, and organizations to adopt the cultural nuances required for effective postmortems. This guide will teach you how to build a culture of continuous learning, the most important components to include in your analysis, and how to conduct effective postmortem meetings.
Having retrospectives on a regular basis is one way for your team to learn what they’re doing right, where they can improve, how to avoid making the same mistakes again and again, and, most importantly, how to critically think about how they’re working together. Well-designed retrospectives allow teams to iteratively improve their end product and collaboration process.
The concept of having retrospectives is well known in the technology industry, but it can be difficult for newer individuals, teams, and organizations to adopt the mindset required to execute effective for effective retrospectives. This guide will teach you how to build a culture of continuous improvement, the most important components to include in your retrospectives, and how to conduct effective retrospectives.
for free. You
can too.