Blood-curdling tales of microservices misadventure devops dread grisly governance Holly Cummins IBM Garage @holly_cummins
A presentation at Software Circus in October 2020 in by Holly Cummins
Blood-curdling tales of microservices misadventure devops dread grisly governance Holly Cummins IBM Garage @holly_cummins
I’m a consultant with the IBM Garage. These are my scary stories #IBMGarage @holly_cummins
is this thing on? http://sli.do #L750 #IBMGarage @holly_cummins
doom! the murky goal
what problem are we trying to solve? #IBMGarage @holly_cummins
doom! microservices envy
#IBMGarage @holly_cummins
we need to microservices #IBMGarage @holly_cummins
microservices are not the goal #IBMGarage @holly_cummins
microservices are not the goal they are the means #IBMGarage @holly_cummins
“we’re going too slowly. we need to get rid of COBOL and make microservices!” #IBMGarage @holly_cummins
“we’re going too slowly. we need to get rid of COBOL and make microservices!” “… but our release board only meets twice a year.” #IBMGarage @holly_cummins
distributed monolith #IBMGarage @holly_cummins
distributed monolith but without compile-time checking … or guaranteed function execution #IBMGarage @holly_cummins
reasons not to do microservices small team not planning to release independently don’t want complexity of a service mesh - or worse yet, rolling your own domain model doesn’t split nicely #IBMGarage @holly_cummins
doom! cloud-native spaghetti
“each of our microservices has duplicated the same object model … with twenty classes and seventy fields” #IBMGarage @holly_cummins
“every time we touch one microservice, the others break” #IBMGarage @holly_cummins
Microservice #IBMGarage @holly_cummins
Domain Microservice #IBMGarage @holly_cummins
Domain Microservice #IBMGarage @holly_cummins
distributed != decoupled #IBMGarage @holly_cummins
doom! microservices ops mayhem
doom! microservices ops mayhem
do you know how to operate these things? (there are quite a few of them) #IBMGarage @holly_cummins
do you know how to operate these things? (there are quite a few of them) #IBMGarage @holly_cummins
observability #IBMGarage @holly_cummins
observability #IBMGarage @holly_cummins
SRE #IBMGarage @holly_cummins
SRE #IBMGarage @holly_cummins
doom! the ‘someday’ automation
“our ops isn’t automated” #IBMGarage @holly_cummins
“our tests aren’t automated” #IBMGarage @holly_cummins
“we don’t know if our code works” #IBMGarage @holly_cummins
“we don’t know if our code works” #IBMGarage @holly_cummins
microservices need automated integration tests #IBMGarage @holly_cummins
microservices need automated contract tests #IBMGarage @holly_cummins
the rotting automation #IBMGarage @holly_cummins
#IBMGarage @holly_cummins
“oh yes, that build has been broken for a few weeks…” #IBMGarage @holly_cummins
“we don’t know when the build is broken” #IBMGarage @holly_cummins
let’s talk about your build http://sli.do #L750 #IBMGarage @holly_cummins
doom! the not-actually-continuous continuous integration and continuous deployment
“we have a CI/CD” #IBMGarage @holly_cummins
“we have a CI/CD” #IBMGarage @holly_cummins
CI/CD is something you do not a tool you buy #IBMGarage @holly_cummins
“i’ll merge my branch into our CI next week” #IBMGarage @holly_cummins
“CI/CD … CI/CD … CI/CD … we release every six months … CI/CD …. ” #IBMGarage @holly_cummins
what is CD? http://sli.do #L750 #IBMGarage @holly_cummins
continuous. I don’t think that word means what you think it means. #IBMGarage @holly_cummins
how do you do continuous? http://sli.do #L750 #IBMGarage @holly_cummins
how do you CD? http://sli.do #L750 #IBMGarage @holly_cummins
“we can’t ship until we have more confidence in the quality” #IBMGarage @holly_cummins
doom! the software crypt
“we can’t actually release this.” #IBMGarage @holly_cummins
why? #IBMGarage @holly_cummins
“we’ve scheduled the architecture board review for a month after the project is ready to ship” #IBMGarage @holly_cummins
“we can’t release this microservice… we deploy all our microservices at the same time.” #IBMGarage @holly_cummins
why? oh yes, we don’t know if they work #IBMGarage @holly_cummins
“we can’t ship until every feature is complete” #IBMGarage @holly_cummins
what’s the point of architecture that can go faster, if you don’t go faster? #IBMGarage @holly_cummins
feedback is good engineering #IBMGarage @holly_cummins
deferred wiring #IBMGarage @holly_cummins
feature flags #IBMGarage @holly_cummins
a/b testing canary deploys #IBMGarage @holly_cummins
doom! the lockeddown totally rigid inflexible un-cloudy cloud
“this provisioning software is broken” #IBMGarage @holly_cummins
10 minute provision-time what we sold “this provisioning software is broken” #IBMGarage @holly_cummins
what the client thought they’d got 10 minute provision-time what we sold 3 month provisiontime “this provisioning software is broken” #IBMGarage @holly_cummins
what the client thought they’d got 10 minute provision-time the reason 3 month provisiontime 84-step pre-approval process what we sold “this provisioning software is broken” #IBMGarage @holly_cummins
#IBMGarage @holly_cummins
governance #IBMGarage @holly_cummins
#IBMGarage @holly_cummins
Provider A Provider B “we’re going to change cloud provider #IBMGarage @holly_cummins
Provider A Provider B “we’re going to change cloud provider to fix our procurement process!” #IBMGarage @holly_cummins
Provider A Provider B “we’re going to change cloud provider to fix our procurement process!” #IBMGarage @holly_cummins
#IBMGarage @holly_cummins
“we’ve configured our network! #IBMGarage @holly_cummins
“we’ve configured our network! you can either access the cloud servers … or access jira. #IBMGarage @holly_cummins
“we’ve configured our network! you can either access the cloud servers … or access jira. to access both you’d need two machines.” #IBMGarage @holly_cummins
“it takes us a week to start coding.” #IBMGarage @holly_cummins
“it takes us a week to start coding.” “two days to get a repo … two days to get a pipeline …” #IBMGarage @holly_cummins
there is a cost: developers flee #IBMGarage @holly_cummins
doom! the mystery money pit
the cloud makes it so easy to provision hardware. IBM Garage @holly_cummins
that doesn’t mean the hardware is free. IBM Garage @holly_cummins
or useful. IBM Garage @holly_cummins
zombie workload #IBMGarage @holly_cummins
2017 survey 25% of 16,000 servers doing no useful work #IBMGarage @holly_cummins
2017 survey 25% of 16,000 servers doing no useful work #IBMGarage @holly_cummins
Hey boss, I created a Kubernetes cluster. #IBMGarage @holly_cummins
Hey boss, I created a Kubernetes cluster. I forgot it for 2 months. #IBMGarage @holly_cummins
Hey boss, I created a Kubernetes cluster. I forgot it for 2 months. … and it’s £1000 a month. #IBMGarage @holly_cummins
“we have 28 cloud instances. or maybe it’s 35.” IBM Garage @holly_cummins
“we have no idea how much we’re spending on cloud.” IBM Garage @holly_cummins
finops multicloud management IBM Garage @holly_cummins
@holly_cummins
