Modern Continuous Delivery

Modern Continuous Delivery A JOURNEY IN 4 ACTS @kmugrage https://www.gocd.org/

Act I UNITED KINGDOM – 2005 @kmugrage https://www.gocd.org/

We knew everything Signed Agile Manifesto Defined Continuous Integration Created the first (or second) CI server Created Selenium @kmugrage https://www.gocd.org/

Java – Write once, run anywhere Developed a system on Windows laptops to be deployed to a Solaris cluster Did all of the right Continuous Integration things One small issue… @kmugrage https://www.gocd.org/

It didn’t work in production @kmugrage https://www.gocd.org/

Conan The Deployer Massive shell script Automated deployment to a cluster after every successful CI run Deployment became a non-issue @kmugrage https://www.gocd.org/

The Output Jez Humble, Dan North and Chris Read write “The Deployment Product Line” Jez Humble creates GoCD Jez Humble and David Farley release the book “Continuous Delivery” @kmugrage https://www.gocd.org/

Act II UNITED KINGDOM – 2000 @kmugrage https://www.gocd.org/

Fairly typical architecture Frontend Edge Load Balancer Web Application Web Application Application Servers Web Application Web Application Web Application MSMQ Web Application Biztalk Customer Service Application Customer Service Application Business Servers Business Servers Business Servers Distributed Cache Session DB @kmugrage Transactional DB Biztalk DB https://www.gocd.org/

Traditional Continuous Delivery Build Test Deploy Get Package git pull Artifact Repository Deploy Test Environment Get Package Artifact Repository Deploy Environment Version Build UI Artifact Repository Smoke Tests Unit Test Service Package Component @kmugrage https://www.gocd.org/

Traditional Continuous Delivery Hand rolled environments Artifact Repository Functional Tests Build Regression Tests Deploy to Stage Deploy to Production Performance Tests @kmugrage https://www.gocd.org/

The good Reproducible builds Generate package once High level of automation Safety net with automated test stages @kmugrage https://www.gocd.org/

The Bad Entire system deployed at once Releases were large Browser tests were flaky Rollbacks were hard @kmugrage https://www.gocd.org/

The Output Sam Newman releases the book “Building Microservices” Kief Morris releases the book “Infrastructure as Code” Created Gauge open source testing framework Created Taiko to combat flaky browser tests @kmugrage https://www.gocd.org/

Act III UNITED STATES – 2019 @kmugrage https://www.gocd.org/

Financial Services Platform Edge Load Balancer Product Domain Apply Domain Service Service Repository Repository Command Handlers Event Handlers Event Store Account Domain Service Repository Command Handlers Event Handlers @kmugrage Event Store E V E N T B U S Command Handlers Event Handlers Event Store Customer Domain Service Repository Command Handlers Event Handlers Event Store https://www.gocd.org/

Modern Pipelines Service A Functional Tests Build Deploy to Stage Deploy to Production Deploy to Stage Deploy to Production Regression Tests Service B Functional Tests Build Regression Tests @kmugrage https://www.gocd.org/

Another option Service A Functional Tests Build Regression Tests Deploy to Stage Deploy to Production Service B Functional Tests Build Regression Tests @kmugrage https://www.gocd.org/

Act IV THE DETAILS @kmugrage https://www.gocd.org/

Traditional Structure Development Teams @kmugrage QA Team Operations Team https://www.gocd.org/

Products over Projects (do the DevOps) Service 1 Service 4 Service 6 Service 5 Service 2 @kmugrage Service 3 https://www.gocd.org/

Develop Build Monitor Test Deploy @kmugrage https://www.gocd.org/

The New Build Artifact git pull Build Unit Test thoughtworks/gocd-server:v18.10 Package Version Docker File @kmugrage https://www.gocd.org/

A modern Continuous delivery pipeline @kmugrage https://www.gocd.org/

Trunk Based Development RELEASE BRANCHES RELEASE 1.1.x HOTFIX CHERRYPICK CHERRYPICK TRUNK COMMIT COMMIT DEVELOPERS @kmugrage SHORT-LIVED DEVELOPMENT BRANCHES https://www.gocd.org/

Feature Toggles @kmugrage https://www.gocd.org/

Develop Build Monitor Test Deploy @kmugrage https://www.gocd.org/

The Test Pyramid UI TESTS Slow, Expensive SERVICE TESTS UNIT TESTS @kmugrage Fast, Cheap https://www.gocd.org/

The Test Pyramid In Context Build Test Deploy Deploy to to Stage Stage Deploy to Production Build Test E2E TESTS Deploy to Stage Slow, Expensive Monitoring Distributed Tracing Fault-injection Testing CONTRACT TESTS COMPONENT TESTS INTEGRATION TESTS UNIT TESTS @kmugrage Fast, Cheap Canary Deployments Blue-Green Deployments A/B Testing https://www.gocd.org/

Develop Build Monitor Test Deploy @kmugrage https://www.gocd.org/

Declarative Deployments Pod Ingress services db Node load balancer volumes Pod Service YAML deployment descriptor Pod Deployment Replica Set Pod Node @kmugrage https://www.gocd.org/

Deployment Strategies Load Balancer V1 V1 Load Balancer V2 V1 V2 Load Balancer V2 V2 V2 V2 Rolling Update Load Balancer V1 V1 V2 Load Balancer Load Balancer V2 V1 V1 V2 V2 V1 V1 75% Blue Green Deployment @kmugrage V2 25% Canary Deployment https://www.gocd.org/

Release DB Changes Out Of Band DB V1 APP V1 DB V2 APP V2 ROLLBACK APP V2 DB migrated to V1 App uses DB V1 DB migrated to V2 App uses DB V2 App rolled back to V1 @kmugrage Time https://www.gocd.org/

Dynamic Environments Build Pipeline Build Test Image metadata Build Image Deploy Stage Deploy Test Image metadata Destroy Deploy Prod Deploy Test Push Image Kubernetes Docker Registry Pull Image namespace: Stage @kmugrage namespace: Prod https://www.gocd.org/

Develop Build Monitor Test Deploy @kmugrage https://www.gocd.org/

Monitoring and observability Metrics Collect & Ship Aggregate & Store Visualize Hardware metrics (CPU, Memory, IO) Metrics Collector App/Business metrics (CPU, Memory, IO) Logs Typical Monitoring Setup @kmugrage https://www.gocd.org/

Monitoring and observability honeycomb.io High cardinality events Observability @kmugrage https://www.gocd.org/

Develop Build Security Monitor Test Deploy @kmugrage https://www.gocd.org/

Develop Build Security Monitor Test Deploy @kmugrage https://www.gocd.org/

Vulnerability Planning Functional Tests Build Deploy to Stage Deploy to Production Regression Tests Static CVE Scan @kmugrage Scan Images Monitor Runtime Continuers Monitor Runtime Continuers https://www.gocd.org/

Secrets Management Use Auth Token To Access Secrets Deliver Secret ID (Auth Token) to the Application Fetch RoleID Continuous Delivery Workflow Trusted Entity @kmugrage Application Server / Container Write RoleID Machine/Container Image ORCHESTRATOR (Chef / Terraform /Kubernetes) https://www.gocd.org/

Epilogue @kmugrage https://www.gocd.org/

In the future Teams have to be structured to own small pieces Docker and Kubernetes are here to stay… until they aren’t You have to test in production. Own it and architect for it Security is everybody’s job @kmugrage https://www.gocd.org/

Resources & Citations http://agilemanifesto.org/ https://martinfowler.com/articles/originalContinuousIntegration.html https://continuousdelivery.com/wp-content/uploads/2011/04/deployment_production_line.pdf https://martinfowler.com/articles/microservice-trade-offs.html https://martinfowler.com/articles/practical-test-pyramid.html https://trunkbaseddevelopment.com/ https://martinfowler.com/articles/feature-toggles.html https://docs.honeycomb.io/learning-about-observability/intro-to-observability/ @kmugrage https://www.gocd.org/

Thank You @kmugrage https://www.gocd.org/