Zen and the Art of Service Ownership

I’m Lilia, a product manager at PagerDuty, where I help customers keep their services healthy, their teams happy. I’m going to talk about how PagerDuty does service ownership, which I have researched a lot, with 3 lessons from zen philosophy, which I have not researched as much.

Set your expectations accordingly.

https://unsplash.com/photos/74TufExdP3Y

Lilia Gutnik // @superlilia

I’m also an incident commander for major incidents. So, I hear a lot of names of services. I recognize most of the acronyms, Greek mythical figures, other naming conventions that didn’t stick.

And during one incident, I heard this one.

https://unsplash.com/photos/loeqHoa1uWY

Daniel Tausis (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

‘Burgundy’

I don’t get it, until someone explains that Burgundy is for reporting.

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

This will bring me to Lesson one. The path to nirvana requires an accurate understanding of the nature of things

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

Ron Burgundy. Anchorman. A reporter.

So with SLA’s and stakeholders swirling around now I also have Will Ferrell and that mustache in my head. Not helpful.

https://www.sky.com/watch/title/programme/35da6c05-1163-4ae7-a41e-b3b8157b18c9

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

Breathe in. Breathe out.

What I need to be successful is an accurate understanding of the nature of things - Or, at least enough of an understanding to have a coherent idea of our service ownership:

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

Service: what is that service, why does it matter. Ownership: what teams owns that service, who can I talk to about it.

Incident Commanders like service names that are easy to understand. Oncall responders love self-describing services.

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

New engineers onboarding to your team. Do you remember joining your team when you were new?

As your company grows, your infrastructure grows, your monolith gets broken down into microservices and now there’s just more of everything: more people, more projects.

https://unsplash.com/photos/KuCGlBXjH_o

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

We wanted to be mindful of how we named our growing number of things; The trade off for fun, inside jokes can be alienating people, adding stress, and wasting time.

You won’t always be there to explain the inside joke. Maybe you’ll come work for Pagerduty, I dunno.

https://unsplash.com/photos/bfJjBskA3gA

Joe Green (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

So, what did we do?

lesson 2, from Pema Chodron: Start where you are.

We didn’t just rename everything all at once. At PagerDuty, we started updating things in place as we went along. Fixing forward, one step at a time.

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

Changes to improve service ownership, like service naming conventions, are more likely to succeed if you fix forward, as new services are built, and new teams are formed.

We started a reliable index of the nature of things actually using PagerDuty: mapping services to teams.

https://unsplash.com/photos/bJhT_8nbUA0

Mikito Tateisi (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

But wait. Izanagi is taking down Pollux and Castor. So, a Japanese deity is battling stars. Greek guys, Nicolas Cage.

Our reliable index still has opaque service names – but it doesn’t make us bad service owners.

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

Zen applied to Service ownership means we don’t try to be perfect.

Lesson 3: We accept that we will evolve all the time, and improve constantly. We try what works for our organization in the stage we’re in now.

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

Now, new services, not only have understandable names, they have a description of their purpose and a tier level. We know how they relate to the business, to understand impacts on customers, or other teams. Naming can still be quirky – teams add emoji’s to their service names, which is pretty cute.

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

And every service now has to be owned by a team, not by any one person. People who built some services have since left - and took a bunch of domain knowledge with them.

We don’t allow services to be owned just one person, or a nobody person.

https://unsplash.com/photos/lbLgFFlADrY

Perry Grone (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

That means teams sometimes own services that didn’t have anything to do with the original service. Which means consider that the service you’re building may be some other team’s responsibility later. How can we help? When we’re on call, leave it better than how you found it. Update a runbook. Get rid of old alerts. Teach other people what you know now. https://unsplash.com/photos/pj-BrFZ9eAA

Pan Xiaozhen (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

Service Ownership is a journey — We’re all building new stuff and patching old stuff, that’s how it goes. As we make decisions about changes we want to make, we fix forward. We try to name things well, we sometimes succeed.

I will never claim that we are perfect. Burgundy still exists. It’s my journey.

https://unsplash.com/photos/qvBYnMuNJ9A

Josue Isai Ramos Figueroa (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

Consider all of the different kinds of teams who will be impacted by our choices. Self-describing service names is just one step towards service ownership that will make your infrastructure approachable for new people, everyone’s on call experience easier, and your incident commander calmer.

https://unsplash.com/photos/My3DsC8yN3Y

Max Bender (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

I’ve met 100’s of people pursuing better service ownership - the happiest ones are fixing forward, starting small, and evolving. We can all learn from our mistakes and our successes.

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

No one no one has it all figured out. There will somehow always be an weirdly named orphaned service, that no one really remembers how to maintain. Everyone’s got a chunk of code that feels too complicated and too sensitive to unravel. It probably made sense at the time, and it was right for the organization you were then.

Stephen Dawson (Unsplash) Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

Start where you are. Examine your services and yourselves with compassion, instead of criticism.

Try something that works now, knowing that things may change.

Take small steps, like a name that’s easy to understand.

“Reporting”.

https://unsplash.com/photos/qwtCeJ5cLYs Photo by Stephen Dawson on Unsplash

Zen and the Art of Service Ownership Lilia Gutnik // @superlilia

Thank you!