It's not Continuous Delivery if you can't deploy right now

A presentation at ThoughtWorks Berlin Meetup in September 2018 in Berlin, Germany by Ken Mugrage

Slide 1

Slide 1

IT’S NOT CONTINUOUS DELIVERY IF YOU CAN’T DEPLOY RIGHT NOW

Slide 2

Slide 2

@kmugrage https://gocd.org/

Slide 3

Slide 3

KEN MUGRAGE THOUGHTWORKS TECHNOLOGY EVANGELIST DEVOPSDAYS GLOBAL ORGANIZER @KMUGRAGE HTTPS://KENMUGRAGE.COM/ WHO AM I?

Slide 4

Slide 4

“DEVOPS: A CULTURE WHERE PEOPLE, REGARDLESS OF TITLE OR BACKGROUND, WORK TOGETHER TO IMAGINE, DEVELOP, DEPLOY AND OPERATE A SYSTEM.” Me @kmugrage https://kenmugrage.com/2017/05/05/my-new-definition-of-devops/

Slide 5

Slide 5

@kmugrage https://gocd.org/

Slide 6

Slide 6

“CONTINUOUS DELIVERY IS THE ABILITY TO GET CHANGES OF ALL TYPES - INCLUDING NEW FEATURES, CONFIGURATION CHANGES, BUG FIXES AND EXPERIMENTS - INTO PRODUCTION, OR INTO THE HANDS OF USERS, SAFELY AND QUICKLY IN A SUSTAINABLE WAY.” Jez Humble @kmugrage https://continuousdelivery.com/

Slide 7

Slide 7

WHY CONTINUOUS DELIVERY?

Slide 8

Slide 8

FULFILL THE PROMISE OF AGILE We follow these principles: Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. @kmugrage https://gocd.org/

Slide 9

Slide 9

PARTIALLY DONE MIGHT BE USEFUL @kmugrage https://gocd.org/

Slide 10

Slide 10

SECURITY THREATS At the time of disclosure, some 17% (around half a million) of the Internet's secure web servers certified by trusted authorities were believed to be vulnerable to the attack, allowing theft of the servers' private keys and users' session cookies and passwords. https://en.wikipedia.org/wiki/Heartbleed @kmugrage https://gocd.org/

Slide 11

Slide 11

MANAGING RISK Knight Capital To be continued…. @kmugrage https://gocd.org/

Slide 12

Slide 12

A PRE-REQUISITE CONTINUOUS INTEGRATION

Slide 13

Slide 13

CONTINUOUS INTEGRATION The ThoughtWorks tech radar recently recommended putting a hold on the tech team anti-pattern, CI Theatre. CI Theatre describes the illusion of practicing continuous integration (CI) while not really practicing it. https://www.gocd.org/2017/05/16/its-not-CI-its-CI-theatre/ @kmugrage https://gocd.org/

Slide 14

Slide 14

CONTINUOUS INTEGRATION In our study only 10% of participants acknowledged that having a CI server was not the same as practicing CI. https://www.gocd.org/2017/05/16/its-not-CI-its-CI-theatre/ @kmugrage https://gocd.org/

Slide 15

Slide 15

CONTINUOUS INTEGRATION @kmugrage https://gocd.org/

Slide 16

Slide 16

CONTINUOUS INTEGRATION @kmugrage https://gocd.org/

Slide 17

Slide 17

CONTINUOUS INTEGRATION @kmugrage https://gocd.org/

Slide 18

Slide 18

FEATURE BRANCHING https://martinfowler.com/bliki/FeatureBranch.html @kmugrage https://gocd.org/

Slide 19

Slide 19

FEATURE BRANCHING https://martinfowler.com/bliki/FeatureBranch.html @kmugrage https://gocd.org/

Slide 20

Slide 20

FEATURE BRANCHING https://martinfowler.com/bliki/FeatureBranch.html @kmugrage https://gocd.org/

Slide 21

Slide 21

DEPLOYING INCOMPLETE WORK

Slide 22

Slide 22

FEATURE TOGGLES https://martinfowler.com/bliki/FeatureToggle.html @kmugrage https://gocd.org/

Slide 23

Slide 23

FEATURE TOGGLES Pete Hodgson - http://martinfowler.com/articles/feature-toggles.html @kmugrage https://gocd.org/

Slide 24

Slide 24

FEATURE TOGGLES Pete Hodgson - http://martinfowler.com/articles/feature-toggles.html @kmugrage https://gocd.org/

Slide 25

Slide 25

CHANGE YOUR ORG TO ENABLE CONTINUOUS DELIVERY

Slide 26

Slide 26

CONTINUOUS DELIVERY PIPELINE @kmugrage https://gocd.org/

Slide 27

Slide 27

TRADITIONAL MODEL Development Teams @kmugrage Testing Team Operations Team https://gocd.org/

Slide 28

Slide 28

“ANY ORGANIZATION THAT DESIGNS A SYSTEM (DEFINED BROADLY) WILL PRODUCE A DESIGN WHOSE STRUCTURE IS A COPY OF THE ORGANIZATION'S COMMUNICATION STRUCTURE.” –Mel Conway @kmugrage http://www.melconway.com/Home/Conways_Law.html

Slide 29

Slide 29

TRADITIONAL MODEL Development Teams @kmugrage QA Team Operations Team https://gocd.org/

Slide 30

Slide 30

RENAMING OPS WON’T SOLVE THE ISSUE Development Teams @kmugrage QA Team DevOps Team https://gocd.org/

Slide 31

Slide 31

CREATING ANOTHER SILO DOESN’T SOLVE SILOS Development Teams QA Team Operations Team DevOps Team @kmugrage https://gocd.org/

Slide 32

Slide 32

PRODUCT TEAMS @kmugrage https://gocd.org/

Slide 33

Slide 33

“YOU BUILD IT, YOU RUN IT” –Werner Vogels, Amazon @kmugrage https://queue.acm.org/detail.cfm?id=1142065

Slide 34

Slide 34

“GIVING DEVELOPERS OPERATIONAL RESPONSIBILITIES HAS GREATLY ENHANCED THE QUALITY OF THE SERVICES, BOTH FROM A CUSTOMER AND A TECHNOLOGY POINT OF VIEW. THE TRADITIONAL MODEL IS THAT YOU TAKE YOUR SOFTWARE TO THE WALL THAT SEPARATES DEVELOPMENT AND OPERATIONS, AND THROW IT OVER AND THEN FORGET ABOUT IT. NOT AT AMAZON. YOU BUILD IT, YOU RUN IT. THIS BRINGS DEVELOPERS INTO CONTACT WITH THE DAY-TO-DAY OPERATION OF THEIR SOFTWARE. IT ALSO BRINGS THEM INTO DAY-TO-DAY CONTACT WITH THE CUSTOMER. THIS CUSTOMER FEEDBACK LOOP IS ESSENTIAL FOR IMPROVING THE QUALITY OF THE SERVICE.” –Werner Vogels, Amazon @kmugrage https://queue.acm.org/detail.cfm?id=1142065

Slide 35

Slide 35

USE MODERN ARCHITECTURES AND TECHNOLOGIES

Slide 36

Slide 36

MONOLITHS CAN BE HARD ▸ All functionality is in one process ▸ Scale by replicating the monolith on multiple servers @kmugrage https://www.thoughtworks.com/insights/blog/microservices-nutshell

Slide 37

Slide 37

DEVELOP SMALLER PIECES MICROSERVICE ARCHITECTURE ▸ Each element of functionality is in a separate service ▸ Scale by distributing these services across servers, replicating as needed @kmugrage https://www.thoughtworks.com/insights/blog/microservices-nutshell

Slide 38

Slide 38

PRODUCT TEAMS Rental Cars Consumer Business to Business Tax Payments Hotels @kmugrage Airlines https://gocd.org/

Slide 39

Slide 39

CAR AS A SERVICE @kmugrage https://gocd.org/

Slide 40

Slide 40

AN EXAMPLE OF PLATFORM AS A SERVICE CLOUD.GOV ▸ Official service of the US Government ▸ 325 required security controls ▸ 269 handled by cloud.gov ▸ 41 shared ▸ 15 handled by customer @kmugrage https://cloud.gov/overview/technology/responsibilities/

Slide 41

Slide 41

THE CLOUD IS JUST SOMEBODY ELSE’S COMPUTER Someone at every conference @kmugrage

Slide 42

Slide 42

PRODUCT TEAMS @kmugrage https://gocd.org/

Slide 43

Slide 43

PRODUCT TEAMS Platform Team @kmugrage Compliance Team Security Team https://gocd.org/

Slide 44

Slide 44

OUR TEAMS Product Team @kmugrage Security Team Compliance Team https://gocd.org/

Slide 45

Slide 45

CONTINUOUS DELIVERY PIPELINE @kmugrage https://gocd.org/

Slide 46

Slide 46

OUR CONTINUOUS DELIVERY PIPELINE UNIT TESTS @kmugrage FUNCTIONAL TESTS DEPLOY STAGING DEPLOY PRODUCTION https://gocd.org/

Slide 47

Slide 47

EXAMPLES OF THINGS WHICH ARE BAD ▸ Deploying insecure software ▸ Deploying non-performant software ▸ Deploying non-complying software ▸ Deploying ineffective software @kmugrage https://gocd.org/

Slide 48

Slide 48

THE PURPOSE OF A CONTINUOUS DELIVERY PIPELINE IS TO KILL A RELEASE CANDIDATE Me, and a lot of other people @kmugrage

Slide 49

Slide 49

OUR CONTINUOUS DELIVERY PIPELINE UNIT TESTS FUNCTIONAL TESTS DEPLOY STAGING DEPLOY PRODUCTION SECURITY TESTS (OWASP, OTHERS) COMPLIANCE TESTS (SERVERSPEC, INSPEC) @kmugrage https://gocd.org/

Slide 50

Slide 50

A REAL LIFE EXAMPLE @kmugrage https://gocd.org/

Slide 51

Slide 51

DEPLOYMENT PATTERNS

Slide 52

Slide 52

WE RECOMMEND STRICTLY USING THE TERM DEPLOYMENT WHEN REFERRING TO THE ACT OF DEPLOYING A CHANGE TO APPLICATION COMPONENTS OR INFRASTRUCTURE. THE TERM RELEASE SHOULD BE USED WHEN A FEATURE CHANGE IS RELEASED TO END USERS. https://www.thoughtworks.com/radar/techniques/decoupling-deployment-from-release @kmugrage

Slide 53

Slide 53

CANARY RELEASE @kmugrage https://gocd.org/

Slide 54

Slide 54

DARK LAUNCHING https://www.facebook.com/notes/facebook-engineering/hammering-usernames/96390263919/ @kmugrage https://gocd.org/

Slide 55

Slide 55

KNIGHT CAPITAL On August 1, 2012, Knight Capital deployed untested software to a production environment which contained an obsolete function. The incident happened due to a technician forgetting to copy the new Retail Liquidity Program (RLP) to one of the eight SMARS computer servers…. …sent millions of child orders, resulting in 4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes. Knight Capital took a pre-tax loss of $440,000,000 @kmugrage https://gocd.org/

Slide 56

Slide 56

SUMMARY ▸ The goal of Continuous Delivery is to make sure your software is always in a deployable state ▸ It’s possible (or even likely) that organizational structure needs to be changed ▸ Smaller pieces are easier to deliver ▸ Don’t skip the “other” kinds of testing @kmugrage https://gocd.org/

Slide 57

Slide 57