A presentation at DevOpsDays Toronto in in Toronto, ON, Canada by Lilia Gutnik
I’m Lilia, a product manager at PagerDuty, where I help customers keep their services healthy, their teams happy. I’m going to talk about how PagerDuty does service ownership, which I have researched a lot, with 3 lessons from zen philosophy, which I have not researched as much.
Set your expectations accordingly.
https://unsplash.com/photos/74TufExdP3Y
Lilia Gutnik // @superlilia
I’m also an incident commander for major incidents. So, I hear a lot of names of services. I recognize most of the acronyms, Greek mythical figures, other naming conventions that didn’t stick.
And during one incident, I heard this one.
https://unsplash.com/photos/loeqHoa1uWY
Daniel Tausis (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
‘Burgundy’
I don’t get it, until someone explains that Burgundy is for reporting.
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
This will bring me to Lesson one. The path to nirvana requires an accurate understanding of the nature of things
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
Ron Burgundy. Anchorman. A reporter.
So with SLA’s and stakeholders swirling around now I also have Will Ferrell and that mustache in my head. Not helpful.
https://www.sky.com/watch/title/programme/35da6c05-1163-4ae7-a41e-b3b8157b18c9
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
Breathe in. Breathe out.
What I need to be successful is an accurate understanding of the nature of things - Or, at least enough of an understanding to have a coherent idea of our service ownership:
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
Service: what is that service, why does it matter. Ownership: what teams owns that service, who can I talk to about it.
Incident Commanders like service names that are easy to understand. Oncall responders love self-describing services.
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
New engineers onboarding to your team. Do you remember joining your team when you were new?
As your company grows, your infrastructure grows, your monolith gets broken down into microservices and now there’s just more of everything: more people, more projects.
https://unsplash.com/photos/KuCGlBXjH_o
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
We wanted to be mindful of how we named our growing number of things; The trade off for fun, inside jokes can be alienating people, adding stress, and wasting time.
You won’t always be there to explain the inside joke. Maybe you’ll come work for Pagerduty, I dunno.
https://unsplash.com/photos/bfJjBskA3gA
Joe Green (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
So, what did we do?
lesson 2, from Pema Chodron: Start where you are.
We didn’t just rename everything all at once. At PagerDuty, we started updating things in place as we went along. Fixing forward, one step at a time.
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
Changes to improve service ownership, like service naming conventions, are more likely to succeed if you fix forward, as new services are built, and new teams are formed.
We started a reliable index of the nature of things actually using PagerDuty: mapping services to teams.
https://unsplash.com/photos/bJhT_8nbUA0
Mikito Tateisi (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
But wait. Izanagi is taking down Pollux and Castor. So, a Japanese deity is battling stars. Greek guys, Nicolas Cage.
Our reliable index still has opaque service names – but it doesn’t make us bad service owners.
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
Zen applied to Service ownership means we don’t try to be perfect.
Lesson 3: We accept that we will evolve all the time, and improve constantly. We try what works for our organization in the stage we’re in now.
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
Now, new services, not only have understandable names, they have a description of their purpose and a tier level. We know how they relate to the business, to understand impacts on customers, or other teams. Naming can still be quirky – teams add emoji’s to their service names, which is pretty cute.
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
And every service now has to be owned by a team, not by any one person. People who built some services have since left - and took a bunch of domain knowledge with them.
We don’t allow services to be owned just one person, or a nobody person.
https://unsplash.com/photos/lbLgFFlADrY
Perry Grone (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
That means teams sometimes own services that didn’t have anything to do with the original service. Which means consider that the service you’re building may be some other team’s responsibility later. How can we help? When we’re on call, leave it better than how you found it. Update a runbook. Get rid of old alerts. Teach other people what you know now. https://unsplash.com/photos/pj-BrFZ9eAA
Pan Xiaozhen (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
Service Ownership is a journey — We’re all building new stuff and patching old stuff, that’s how it goes. As we make decisions about changes we want to make, we fix forward. We try to name things well, we sometimes succeed.
I will never claim that we are perfect. Burgundy still exists. It’s my journey.
https://unsplash.com/photos/qvBYnMuNJ9A
Josue Isai Ramos Figueroa (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
Consider all of the different kinds of teams who will be impacted by our choices. Self-describing service names is just one step towards service ownership that will make your infrastructure approachable for new people, everyone’s on call experience easier, and your incident commander calmer.
https://unsplash.com/photos/My3DsC8yN3Y
Max Bender (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
I’ve met 100’s of people pursuing better service ownership - the happiest ones are fixing forward, starting small, and evolving. We can all learn from our mistakes and our successes.
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
No one no one has it all figured out. There will somehow always be an weirdly named orphaned service, that no one really remembers how to maintain. Everyone’s got a chunk of code that feels too complicated and too sensitive to unravel. It probably made sense at the time, and it was right for the organization you were then.
Stephen Dawson (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
Start where you are. Examine your services and yourselves with compassion, instead of criticism.
Try something that works now, knowing that things may change.
Take small steps, like a name that’s easy to understand.
“Reporting”.
https://unsplash.com/photos/qwtCeJ5cLYs Photo by Stephen Dawson on Unsplash
Zen and the Art of Service Ownership Lilia Gutnik // @superlilia
Thank you!
How does PagerDuty practice devops service ownership? By embracing concepts from Zen philosophy: start where you are, evolve constantly, and fail often. We’ve learned how to help people be successful service owners and we can teach you, too.
Here’s what was said about this presentation on social media.
for free. You
can too.