Blood-curdling tales of microservices misadventure, devops dread, and grisly governance

Blood-curdling tales of microservices misadventure devops dread grisly governance Holly Cummins IBM Garage @holly_cummins

I’m a consultant with the IBM Garage. These are my scary stories #IBMGarage @holly_cummins

is this thing on? http://sli.do #L750 #IBMGarage @holly_cummins

doom! the murky goal

what problem are we trying to solve? #IBMGarage @holly_cummins

doom! microservices envy

#IBMGarage @holly_cummins

we need to microservices #IBMGarage @holly_cummins

microservices are not the goal #IBMGarage @holly_cummins

microservices are not the goal they are the means #IBMGarage @holly_cummins

“we’re going too slowly. we need to get rid of COBOL and make microservices!” #IBMGarage @holly_cummins

“we’re going too slowly. we need to get rid of COBOL and make microservices!” “… but our release board only meets twice a year.” #IBMGarage @holly_cummins

distributed monolith #IBMGarage @holly_cummins

distributed monolith but without compile-time checking … or guaranteed function execution #IBMGarage @holly_cummins

reasons not to do microservices small team not planning to release independently don’t want complexity of a service mesh - or worse yet, rolling your own domain model doesn’t split nicely #IBMGarage @holly_cummins

doom! cloud-native spaghetti

“each of our microservices has duplicated the same object model … with twenty classes and seventy fields” #IBMGarage @holly_cummins

“every time we touch one microservice, the others break” #IBMGarage @holly_cummins

Microservice #IBMGarage @holly_cummins

Domain Microservice #IBMGarage @holly_cummins

distributed != decoupled #IBMGarage @holly_cummins

doom! microservices ops mayhem

do you know how to operate these things? (there are quite a few of them) #IBMGarage @holly_cummins

observability #IBMGarage @holly_cummins

SRE #IBMGarage @holly_cummins

doom! the ‘someday’ automation

“our ops isn’t automated” #IBMGarage @holly_cummins

“our tests aren’t automated” #IBMGarage @holly_cummins

“we don’t know if our code works” #IBMGarage @holly_cummins

microservices need automated integration tests #IBMGarage @holly_cummins

microservices need automated contract tests #IBMGarage @holly_cummins

the rotting automation #IBMGarage @holly_cummins

#IBMGarage @holly_cummins

“oh yes, that build has been broken for a few weeks…” #IBMGarage @holly_cummins

“we don’t know when the build is broken” #IBMGarage @holly_cummins

let’s talk about your build http://sli.do #L750 #IBMGarage @holly_cummins

doom! the not-actually-continuous continuous integration and continuous deployment

“we have a CI/CD” #IBMGarage @holly_cummins

CI/CD is something you do not a tool you buy #IBMGarage @holly_cummins

“i’ll merge my branch into our CI next week” #IBMGarage @holly_cummins

“CI/CD … CI/CD … CI/CD … we release every six months … CI/CD …. ” #IBMGarage @holly_cummins

what is CD? http://sli.do #L750 #IBMGarage @holly_cummins

continuous. I don’t think that word means what you think it means. #IBMGarage @holly_cummins

how do you do continuous? http://sli.do #L750 #IBMGarage @holly_cummins

how do you CD? http://sli.do #L750 #IBMGarage @holly_cummins

“we can’t ship until we have more confidence in the quality” #IBMGarage @holly_cummins

doom! the software crypt

“we can’t actually release this.” #IBMGarage @holly_cummins

why? #IBMGarage @holly_cummins

“we’ve scheduled the architecture board review for a month after the project is ready to ship” #IBMGarage @holly_cummins

“we can’t release this microservice… we deploy all our microservices at the same time.” #IBMGarage @holly_cummins

why? oh yes, we don’t know if they work #IBMGarage @holly_cummins

“we can’t ship until every feature is complete” #IBMGarage @holly_cummins

what’s the point of architecture that can go faster, if you don’t go faster? #IBMGarage @holly_cummins

feedback is good engineering #IBMGarage @holly_cummins

deferred wiring #IBMGarage @holly_cummins

feature flags #IBMGarage @holly_cummins

a/b testing canary deploys #IBMGarage @holly_cummins

doom! the lockeddown totally rigid inflexible un-cloudy cloud

“this provisioning software is broken” #IBMGarage @holly_cummins

10 minute provision-time what we sold “this provisioning software is broken” #IBMGarage @holly_cummins

what the client thought they’d got 10 minute provision-time what we sold 3 month provisiontime “this provisioning software is broken” #IBMGarage @holly_cummins

what the client thought they’d got 10 minute provision-time the reason 3 month provisiontime 84-step pre-approval process what we sold “this provisioning software is broken” #IBMGarage @holly_cummins

#IBMGarage @holly_cummins

governance #IBMGarage @holly_cummins

#IBMGarage @holly_cummins

Provider A Provider B “we’re going to change cloud provider #IBMGarage @holly_cummins

Provider A Provider B “we’re going to change cloud provider to fix our procurement process!” #IBMGarage @holly_cummins

#IBMGarage @holly_cummins

“we’ve configured our network! #IBMGarage @holly_cummins

“we’ve configured our network! you can either access the cloud servers … or access jira. #IBMGarage @holly_cummins

“we’ve configured our network! you can either access the cloud servers … or access jira. to access both you’d need two machines.” #IBMGarage @holly_cummins

“it takes us a week to start coding.” #IBMGarage @holly_cummins

“it takes us a week to start coding.” “two days to get a repo … two days to get a pipeline …” #IBMGarage @holly_cummins

there is a cost: developers flee #IBMGarage @holly_cummins

doom! the mystery money pit

the cloud makes it so easy to provision hardware. IBM Garage @holly_cummins

that doesn’t mean the hardware is free. IBM Garage @holly_cummins

or useful. IBM Garage @holly_cummins

zombie workload #IBMGarage @holly_cummins

2017 survey 25% of 16,000 servers doing no useful work #IBMGarage @holly_cummins

Hey boss, I created a Kubernetes cluster. #IBMGarage @holly_cummins

Hey boss, I created a Kubernetes cluster. I forgot it for 2 months. #IBMGarage @holly_cummins

Hey boss, I created a Kubernetes cluster. I forgot it for 2 months. … and it’s £1000 a month. #IBMGarage @holly_cummins

“we have 28 cloud instances. or maybe it’s 35.” IBM Garage @holly_cummins

“we have no idea how much we’re spending on cloud.” IBM Garage @holly_cummins

finops multicloud management IBM Garage @holly_cummins

@holly_cummins