A presentation at DevOps Enterprise Summit London 2019 in in London, UK by Holly Cummins
Tales from the DevOps Transformation Trenches yes, you (still) need to start with culture, not containers Austin Copenhagen Dubai London Madrid Melbourne Munich New York Nice Holly Cummins IBM Garage @holly_cummins Raleigh San Francisco São Paulo Singapore Tokyo Toronto
#IBMGarage @holly_cummins
#IBMGarage @holly_cummins
#IBMGarage @holly_cummins
#IBMGarage @holly_cummins
#IBMGarage @holly_cummins
#IBMGarage @holly_cummins
hi. i’m a consultant. #IBMGarage @holly_cummins
these are my scary stories #IBMGarage @holly_cummins
how to fail at devops #IBMGarage @holly_cummins
“this is our devops team” #IBMGarage @holly_cummins
“this is our devops team” “… last year we called them the build team.” #IBMGarage @holly_cummins
containers will not fix your broken devops culture 8
even kubernetes will not fix your broken devops culture 9
“we’re going too slowly. we need to get rid of COBOL and make microservices!” #IBMGarage @holly_cummins
“we’re going too slowly. we need to get rid of COBOL and make microservices!” “… but our release board only meets twice a year.” #IBMGarage @holly_cummins
https://hackernoon.com/8-devops-trends-to-be-aware-of-in-2019-b4232ac8f351 #IBMGarage @holly_cummins
https://hackernoon.com/8-devops-trends-to-be-aware-of-in-2019-b4232ac8f351 #IBMGarage @holly_cummins
“every time we change code, something breaks” #IBMGarage @holly_cummins
distributed monolith #IBMGarage @holly_cummins
distributed monolith but without compile-time checking #IBMGarage @holly_cummins
just because a system runs across 6 containers doesn’t mean it’s decoupled #IBMGarage @holly_cummins
#IBMGarage @holly_cummins
mars climate explorer #IBMGarage @holly_cummins
for clarity: this wasn’t a client of mine. other people’s trenches #IBMGarage @holly_cummins
Courtesy NASA/ JPL-Caltech #IBMGarage @holly_cummins
#IBMGarage @holly_cummins
#IBMGarage @holly_cummins
metric units
metric units imperial units
metric units imperial units distributing did not help
microservices need consumer-driven contract tests #IBMGarage @holly_cummins
Cluster + Ariane 5 $370 million loss https://en.wikipedia.org/wiki/Cluster_(spacecraft) #IBMGarage @holly_cummins
Cluster + Ariane 5 $370 million loss https://en.wikipedia.org/wiki/Cluster_(spacecraft) #IBMGarage @holly_cummins
#IBMGarage @holly_cummins
they tested it … #IBMGarage @holly_cummins
they tested it … but stubbed out one component. #IBMGarage @holly_cummins
they tested it … but stubbed out one component. that component was the one that broke. #IBMGarage @holly_cummins
“Had we done end-to-end testing, we believe this error would have been caught.” Arthur Stephenson Chief Investigator #IBMGarage @holly_cummins
microservices need automated integration tests #IBMGarage @holly_cummins
“we have a CI/CD” #IBMGarage @holly_cummins
CI/CD is something you do, it’s not a tool you buy #IBMGarage @holly_cummins
“i’ll merge my branch into our CI next week” #IBMGarage @holly_cummins
“CI/CD … CI/CD … CI/CD … we release every six months … CI/CD …. ” #IBMGarage @holly_cummins
continuous. I don’t think that word means what you think it means. #IBMGarage @holly_cummins
how often should you push to master? #IBMGarage @holly_cummins
how often should you push to master? integrate? #IBMGarage @holly_cummins
how often should you push to master? integrate? every character #IBMGarage @holly_cummins
how often should you push to master? integrate? every character actually continuous … but stupid #IBMGarage @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) actually continuous … but stupid #IBMGarage @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) actually continuous … but stupid #IBMGarage @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day actually continuous … but stupid #IBMGarage @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day once a week actually continuous … but stupid #IBMGarage @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day once a week once a month actually continuous … but stupid #IBMGarage @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day once a week once a month once every six months actually continuous … but stupid #IBMGarage @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day once a week once a month once every six months actually continuous … but stupid #IBMGarage trunk-based development @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day ok actually continuous … but stupid #IBMGarage once a week once a month once every six months trunk-based development @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day ok actually continuous … but stupid #IBMGarage bad once a week once a month once every six months trunk-based development @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day ok once a week once a month once every six months bad bad actually continuous … but stupid #IBMGarage trunk-based development @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day ok once a week once a month once every six months bad bad seriously? actually continuous … but stupid #IBMGarage trunk-based development @holly_cummins
how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day ok once a week once a month once every six months bad bad my favourite actually continuous … but stupid #IBMGarage seriously? trunk-based development @holly_cummins
how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter #IBMGarage once every two years @holly_cummins
deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter #IBMGarage once every two years @holly_cummins
deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter once every two years (need a good handle on feature flags) #IBMGarage @holly_cummins
deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok once every two years (need a good handle on feature flags) #IBMGarage @holly_cummins
deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok (need a good handle on feature flags) #IBMGarage once every two years oldschool @holly_cummins
deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok once every two years sigh (need a good handle on feature flags) #IBMGarage oldschool @holly_cummins
deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok ok once every two years sigh (need a good handle on feature flags) #IBMGarage oldschool @holly_cummins
deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok ok once every two years sigh hardcore (need a good handle on feature flags) #IBMGarage oldschool @holly_cummins
deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok ok once every two years sigh hardcore (need a good handle on feature flags) #IBMGarage my favourite oldschool @holly_cummins
how often should you test in staging? #IBMGarage @holly_cummins
how often should you test in staging? deliver? #IBMGarage @holly_cummins
how often should you test in staging? deliver? every push my favourite #IBMGarage @holly_cummins
“we can’t actually release this.” #IBMGarage @holly_cummins
why? #IBMGarage @holly_cummins
what’s stopping more frequent deploys? #IBMGarage @holly_cummins
“we can’t release this microservice… we deploy all our microservices at the same time.” #IBMGarage @holly_cummins
“it looks like it’s complete … but nothing works if you click on it.” #IBMGarage @holly_cummins
front-end integration layer back-end #IBMGarage @holly_cummins
user story front-end integration layer back-end #IBMGarage @holly_cummins
user story #IBMGarage front-end frontend integration layer int. layer back-end backend @holly_cummins
user story front-end frontend integration layer back-end #IBMGarage @holly_cummins
user story front-end integration layer back-end #IBMGarage @holly_cummins
user story front-end integration layer back-end #IBMGarage backend @holly_cummins
user story front-end #IBMGarage integration layer int. layer back-end backend @holly_cummins
✓ it works by the time anyone sees it user story front-end frontend integration layer int. layer back-end backend stakeholders need to be careful what they incentivise #IBMGarage @holly_cummins
vertical slices #IBMGarage @holly_cummins
back-out development #IBMGarage @holly_cummins
back-out back-first development development #IBMGarage @holly_cummins
deferred wiring #IBMGarage @holly_cummins
feature flags #IBMGarage @holly_cummins
“we can’t ship until every feature is complete” #IBMGarage @holly_cummins
but why? #IBMGarage @holly_cummins
“users won’t find it compelling enough if we release now” #IBMGarage @holly_cummins
if you’re not embarrassed by your first release it was too late - Reid Hoffman #IBMGarage @holly_cummins
lean #IBMGarage @holly_cummins
“we only get one chance to get it right” #IBMGarage @holly_cummins
the ariadne failed in 36 seconds you can’t a/b test a $370 million rocket #IBMGarage @holly_cummins
we think we’re here one chance #IBMGarage @holly_cummins
we think we’re here one chance #IBMGarage brand damage @holly_cummins
we think we’re here market failure (indifference) one chance #IBMGarage brand damage @holly_cummins
we think we’re here market failure (indifference) continuous improvement delights growing user base one chance #IBMGarage brand damage @holly_cummins
we think we’re here market failure (indifference) a/b testing continuous improvement delights growing user base one chance #IBMGarage brand damage @holly_cummins
could we be here? we think we’re here market failure (indifference) a/b testing continuous improvement delights growing user base one chance #IBMGarage brand damage @holly_cummins
feedback is good engineering #IBMGarage @holly_cummins
they often couldn’t see the explorer #IBMGarage @holly_cummins
“but our change control process …” #IBMGarage @holly_cummins
“this provisioning software is broken” #IBMGarage @holly_cummins
10 minute provision-time what we sold “this provisioning software is broken” #IBMGarage @holly_cummins
what the client thought they’d got 10 minute provision-time what we sold 3 month provisiontime “this provisioning software is broken” #IBMGarage @holly_cummins
what the client thought they’d got 10 minute provision-time the reason 3 month provisiontime 84-step pre-approval process what we sold “this provisioning software is broken” #IBMGarage @holly_cummins
“we’ve scheduled the architecture board review for a month after the project ships” #IBMGarage @holly_cummins
does the process add value? #IBMGarage @holly_cummins
#IBMGarage @holly_cummins
navigators warned something was wrong #IBMGarage @holly_cummins
navigators warned something was wrong they didn’t fill in the right form #IBMGarage @holly_cummins
navigators warned something was wrong they didn’t fill in the right form so nothing was done #IBMGarage @holly_cummins
“we can’t ship until we have more confidence in the quality” #IBMGarage @holly_cummins
“we can’t ship until we have more confidence in the quality” you can fix that #IBMGarage @holly_cummins
“this is the test team … who don’t have the skills to automate their tests.” #IBMGarage @holly_cummins
“our tests aren’t automated” #IBMGarage @holly_cummins
“our tests aren’t automated” #IBMGarage @holly_cummins
“we don’t know if our code currently works” #IBMGarage @holly_cummins
“we don’t know if our code currently works” #IBMGarage @holly_cummins
“it costs too much to release” #IBMGarage @holly_cummins
“it costs too much to release”you can fix that #IBMGarage @holly_cummins
not a good CI/CD indicator a good CI/CD indicator “we don’t know when the build is broken” #IBMGarage @holly_cummins
get the pipeline status into the physical spaces #IBMGarage @holly_cummins
“only Bob can change Jenkins” #IBMGarage @holly_cummins
#IBMGarage @holly_cummins
“oh yes, that build has been broken for a few weeks…” #IBMGarage @holly_cummins
judge judge judge #IBMGarage @holly_cummins
modern devops toolchains and processes reflect cloud native apps and cultural transformation many, single-tenant toolchains #IBMGarage hybrid and multicloud toolchains and deployments toolchains support lean delivery processes and business agility @holly_cummins
heritage devops toolchains and processes reflect heritage apps and cultural inertia shared, multi-tenant toolchain “backbone” #IBMGarage on-premise automation tools release management and dependency coordination are hard @holly_cummins
“you’ll be coding on the mainframe” #IBMGarage @holly_cummins
#IBMGarage @holly_cummins
#IBMGarage @holly_cummins
this can get tiring #IBMGarage @holly_cummins
transformation endurance #IBMGarage @holly_cummins
remember the why #IBMGarage @holly_cummins
® @holly_cummins IBM Cloud Garage
Here’s what was said about this presentation on social media.
for free. You
can too.