Matt Davis SRE Advocate Blameless Observability, Incident Response, & Common Ground that Binds
A presentation at CMG ObservabilityCon in September 2022 in by matt davis
Matt Davis SRE Advocate Blameless Observability, Incident Response, & Common Ground that Binds
How Complex Systems Fail Dr. Richard Cook 12. Human practitioners are the adaptable element of complex systems. Practitioners and first line management actively adapt the system to maximize production and minimize accidents.
Common Ground & Joint Activity
Joint Activity Parties intend to work together, not in parallel. ~ Their work is interdependent. ~ All are committed to common goals. ~ Collaborative choreography is achieved through effective coordination. Interpredictability [ Reciprocity ] Directability [ Reframing ] Common Ground [ Group Intuition ]
Common Ground qualities of grounding A process of communicating, testing, updating, tailoring, & repairing mutual understandings & mental models, consisting of: - Initial Conditions - Status of what has transpired - Changes of knowledge since we started -
Common Ground types of knowledge, beliefs, assumptions ~ Each participant’s Role ~ ~ Routines the team can handle ~ ~ Expertise of each person ~ ~ Each person’s Stance: their perception of production pressure, level of fatigue, cognitive weight, competing priorities ~
Common Ground key aspects for incidents A. We commit to continually inspect and adjust Common Ground as we work towards remediation. B. We share the types of knowledge, beliefs, and assumptions at work. C. We recognize coordination signals.
The Response Trio
Incident Command adaptive choreography
Incident Command responsibilities
Incident Command adaptive choreography Command remains flexible to an emerging situation and guides the response to remediation, giving Problem Solvers autonomy to selfcoordinate and make local decisions as needed.
for the Interoperation of Distributed Software, Incidents need: Machine Observability for the Reciprocity & Interpredictability of People, Incidents also need: Human Observability
The Human Observability of Incident Command
Dynamics of Reciprocity The ability to Dr. Laura Maguire “look in and listen in” has been widely documented as a benefit to smooth coordination. - Managing the Hidden Costs of Coordination, ACM Queue
Human Observability support common grounding Listen. Get the big picture by asking others their perspective. Be attentive to peoples’ ability to respond while keeping a regular tempo with check-ins. Update. When new people join, give them a status. When a mitigation is made, announce it. Feel like you are overcommunicating. Delegate when you need help updating. Guide. Get multiple perspectives for making decisions. Politely keep people on-topic. Suggest next steps or alternatives to engage minds.
Human Observability support common grounding Monitor. Recognize signals like: questions around confusion, side conversations, huddles without updates, lack of response from a Role, hypothesis testing goes too long, a Problem Solver becoming fatigued. Repair. Get knowledge flowing by asking questions. Redirect threads to the main chat. Delegate a Scribe. Pull participants together for a no-work status chat/meeting. Practice. Use dedicated collaboration sessions outside of incidents to build personal relationships and group intuition. Provide a safe, unpressurized space for people to feel vulnerable with their peers.
Human Observability takeaways for incidents A. Share and update mental models to support common goals. B. Maintain transparency to support reciprocity and signals. C. Be flexible enough to change course and recalibrate when breakdowns occur.
References 1. Cook, Richard. How Complex Systems Fail. https://how.complexsystems.fail. 2. Klein, G., Feltovich, P. J., Bradshaw, J. M., Woods, D. D. 2004. Common ground and coordination in joint activity. http://jeffreymbradshaw.net/publications/ Common_Ground_Single.pdf. 3. Maguire, Laura. Managing the Hidden Costs of Coordination. https://queue.acm.org/detail.cfm?id=3380779.
Q&A matt davis @dtauvdiodr
