Tales From The DevOps Transformation Trenches

Tales from the DevOps Transformation Trenches yes, you (still) need to start with culture, not containers Austin Copenhagen Dubai London Madrid Melbourne Munich New York Nice Holly Cummins IBM Garage @holly_cummins Raleigh San Francisco São Paulo Singapore Tokyo Toronto

#IBMGarage @holly_cummins

hi. i’m a consultant. #IBMGarage @holly_cummins

these are my scary stories #IBMGarage @holly_cummins

how to fail at devops #IBMGarage @holly_cummins

“this is our devops team” #IBMGarage @holly_cummins

“this is our devops team” “… last year we called them the build team.” #IBMGarage @holly_cummins

containers will not fix your broken devops culture 8

even kubernetes will not fix your broken devops culture 9

“we’re going too slowly. we need to get rid of COBOL and make microservices!” #IBMGarage @holly_cummins

“we’re going too slowly. we need to get rid of COBOL and make microservices!” “… but our release board only meets twice a year.” #IBMGarage @holly_cummins

https://hackernoon.com/8-devops-trends-to-be-aware-of-in-2019-b4232ac8f351 #IBMGarage @holly_cummins

“every time we change code, something breaks” #IBMGarage @holly_cummins

distributed monolith #IBMGarage @holly_cummins

distributed monolith but without compile-time checking #IBMGarage @holly_cummins

just because a system runs across 6 containers doesn’t mean it’s decoupled #IBMGarage @holly_cummins

#IBMGarage @holly_cummins

mars climate explorer #IBMGarage @holly_cummins

for clarity: this wasn’t a client of mine. other people’s trenches #IBMGarage @holly_cummins

Courtesy NASA/ JPL-Caltech #IBMGarage @holly_cummins

#IBMGarage @holly_cummins

metric units

metric units imperial units

metric units imperial units distributing did not help

microservices need consumer-driven contract tests #IBMGarage @holly_cummins

Cluster + Ariane 5 $370 million loss https://en.wikipedia.org/wiki/Cluster_(spacecraft) #IBMGarage @holly_cummins

#IBMGarage @holly_cummins

they tested it … #IBMGarage @holly_cummins

they tested it … but stubbed out one component. #IBMGarage @holly_cummins

they tested it … but stubbed out one component. that component was the one that broke. #IBMGarage @holly_cummins

“Had we done end-to-end testing, we believe this error would have been caught.” Arthur Stephenson Chief Investigator #IBMGarage @holly_cummins

microservices need automated integration tests #IBMGarage @holly_cummins

“we have a CI/CD” #IBMGarage @holly_cummins

CI/CD is something you do, it’s not a tool you buy #IBMGarage @holly_cummins

“i’ll merge my branch into our CI next week” #IBMGarage @holly_cummins

“CI/CD … CI/CD … CI/CD … we release every six months … CI/CD …. ” #IBMGarage @holly_cummins

continuous. I don’t think that word means what you think it means. #IBMGarage @holly_cummins

how often should you push to master? #IBMGarage @holly_cummins

how often should you push to master? integrate? #IBMGarage @holly_cummins

how often should you push to master? integrate? every character #IBMGarage @holly_cummins

how often should you push to master? integrate? every character actually continuous … but stupid #IBMGarage @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) actually continuous … but stupid #IBMGarage @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) actually continuous … but stupid #IBMGarage @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day actually continuous … but stupid #IBMGarage @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day once a week actually continuous … but stupid #IBMGarage @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day once a week once a month actually continuous … but stupid #IBMGarage @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day once a week once a month once every six months actually continuous … but stupid #IBMGarage @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day once a week once a month once every six months actually continuous … but stupid #IBMGarage trunk-based development @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day ok actually continuous … but stupid #IBMGarage once a week once a month once every six months trunk-based development @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day ok actually continuous … but stupid #IBMGarage bad once a week once a month once every six months trunk-based development @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day ok once a week once a month once every six months bad bad actually continuous … but stupid #IBMGarage trunk-based development @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day ok once a week once a month once every six months bad bad seriously? actually continuous … but stupid #IBMGarage trunk-based development @holly_cummins

how often should you push to master? integrate? every character every commit (several times an hour) every few commits (several times a day) once a day ok once a week once a month once every six months bad bad my favourite actually continuous … but stupid #IBMGarage seriously? trunk-based development @holly_cummins

how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter #IBMGarage once every two years @holly_cummins

deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter #IBMGarage once every two years @holly_cummins

deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter once every two years (need a good handle on feature flags) #IBMGarage @holly_cummins

deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok once every two years (need a good handle on feature flags) #IBMGarage @holly_cummins

deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok (need a good handle on feature flags) #IBMGarage once every two years oldschool @holly_cummins

deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok once every two years sigh (need a good handle on feature flags) #IBMGarage oldschool @holly_cummins

deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok ok once every two years sigh (need a good handle on feature flags) #IBMGarage oldschool @holly_cummins

deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok ok once every two years sigh hardcore (need a good handle on feature flags) #IBMGarage oldschool @holly_cummins

deploy? how often should you release? every push (many times a day) every user story every epic once a sprint once a quarter ok ok once every two years sigh hardcore (need a good handle on feature flags) #IBMGarage my favourite oldschool @holly_cummins

how often should you test in staging? #IBMGarage @holly_cummins

how often should you test in staging? deliver? #IBMGarage @holly_cummins

how often should you test in staging? deliver? every push my favourite #IBMGarage @holly_cummins

“we can’t actually release this.” #IBMGarage @holly_cummins

why? #IBMGarage @holly_cummins

what’s stopping more frequent deploys? #IBMGarage @holly_cummins

“we can’t release this microservice… we deploy all our microservices at the same time.” #IBMGarage @holly_cummins

“it looks like it’s complete … but nothing works if you click on it.” #IBMGarage @holly_cummins

front-end integration layer back-end #IBMGarage @holly_cummins

user story front-end integration layer back-end #IBMGarage @holly_cummins

user story #IBMGarage front-end frontend integration layer int. layer back-end backend @holly_cummins

user story front-end frontend integration layer back-end #IBMGarage @holly_cummins

user story front-end integration layer back-end #IBMGarage @holly_cummins

user story front-end integration layer back-end #IBMGarage backend @holly_cummins

user story front-end #IBMGarage integration layer int. layer back-end backend @holly_cummins

✓ it works by the time anyone sees it user story front-end frontend integration layer int. layer back-end backend stakeholders need to be careful what they incentivise #IBMGarage @holly_cummins

vertical slices #IBMGarage @holly_cummins

back-out development #IBMGarage @holly_cummins

back-out back-first development development #IBMGarage @holly_cummins

deferred wiring #IBMGarage @holly_cummins

feature flags #IBMGarage @holly_cummins

“we can’t ship until every feature is complete” #IBMGarage @holly_cummins

but why? #IBMGarage @holly_cummins

“users won’t find it compelling enough if we release now” #IBMGarage @holly_cummins

if you’re not embarrassed by your first release it was too late - Reid Hoffman #IBMGarage @holly_cummins

lean #IBMGarage @holly_cummins

“we only get one chance to get it right” #IBMGarage @holly_cummins

the ariadne failed in 36 seconds you can’t a/b test a $370 million rocket #IBMGarage @holly_cummins

we think we’re here one chance #IBMGarage @holly_cummins

we think we’re here one chance #IBMGarage brand damage @holly_cummins

we think we’re here market failure (indifference) one chance #IBMGarage brand damage @holly_cummins

we think we’re here market failure (indifference) continuous improvement delights growing user base one chance #IBMGarage brand damage @holly_cummins

we think we’re here market failure (indifference) a/b testing continuous improvement delights growing user base one chance #IBMGarage brand damage @holly_cummins

could we be here? we think we’re here market failure (indifference) a/b testing continuous improvement delights growing user base one chance #IBMGarage brand damage @holly_cummins

feedback is good engineering #IBMGarage @holly_cummins

they often couldn’t see the explorer #IBMGarage @holly_cummins

“but our change control process …” #IBMGarage @holly_cummins

“this provisioning software is broken” #IBMGarage @holly_cummins

10 minute provision-time what we sold “this provisioning software is broken” #IBMGarage @holly_cummins

what the client thought they’d got 10 minute provision-time what we sold 3 month provisiontime “this provisioning software is broken” #IBMGarage @holly_cummins

what the client thought they’d got 10 minute provision-time the reason 3 month provisiontime 84-step pre-approval process what we sold “this provisioning software is broken” #IBMGarage @holly_cummins

“we’ve scheduled the architecture board review for a month after the project ships” #IBMGarage @holly_cummins

does the process add value? #IBMGarage @holly_cummins

#IBMGarage @holly_cummins

navigators warned something was wrong #IBMGarage @holly_cummins

navigators warned something was wrong they didn’t fill in the right form #IBMGarage @holly_cummins

navigators warned something was wrong they didn’t fill in the right form so nothing was done #IBMGarage @holly_cummins

“we can’t ship until we have more confidence in the quality” #IBMGarage @holly_cummins

“we can’t ship until we have more confidence in the quality” you can fix that #IBMGarage @holly_cummins

“this is the test team … who don’t have the skills to automate their tests.” #IBMGarage @holly_cummins

“our tests aren’t automated” #IBMGarage @holly_cummins

“we don’t know if our code currently works” #IBMGarage @holly_cummins

“it costs too much to release” #IBMGarage @holly_cummins

“it costs too much to release”you can fix that #IBMGarage @holly_cummins

not a good CI/CD indicator a good CI/CD indicator “we don’t know when the build is broken” #IBMGarage @holly_cummins

get the pipeline status into the physical spaces #IBMGarage @holly_cummins

“only Bob can change Jenkins” #IBMGarage @holly_cummins

#IBMGarage @holly_cummins

“oh yes, that build has been broken for a few weeks…” #IBMGarage @holly_cummins

judge judge judge #IBMGarage @holly_cummins

modern devops toolchains and processes reflect cloud native apps and cultural transformation many, single-tenant toolchains #IBMGarage hybrid and multicloud toolchains and deployments toolchains support lean delivery processes and business agility @holly_cummins

heritage devops toolchains and processes reflect heritage apps and cultural inertia shared, multi-tenant toolchain “backbone” #IBMGarage on-premise automation tools release management and dependency coordination are hard @holly_cummins

“you’ll be coding on the mainframe” #IBMGarage @holly_cummins

#IBMGarage @holly_cummins

this can get tiring #IBMGarage @holly_cummins

transformation endurance #IBMGarage @holly_cummins

remember the why #IBMGarage @holly_cummins

® @holly_cummins IBM Cloud Garage