A presentation at COMET SIL - DevOps & Agile, REX et opportunités by Horacio Gonzalez
OVHcloud K8s Team DevOps & Agility in a Product Team Horacio Gonzalez 2020-07-09
Who are we? Introducing myself and introducing OVH OVHcloud
Horacio Gonzalez @LostInBrittany Spaniard lost in Brittany, developer, dreamer and all-around geek Flutter
OVHcloud: A Global Leader 200k Private cloud VMs running 1 Dedicated IaaS Europe 30 Datacenters Own 20Tbps Hosting capacity : 1.3M Physical Servers 360k Servers already deployed Netwok with 35 PoPs
1.3M Customers in 138 Countries
OVHcloud: 4 Universes of Products WebCloud Domain / Email Domain names, DNS, SSL, Redirect Email, Open-Xchange, Exchange Baremetal Cloud VM General Purpose Baremetal SuperPlan T2 >20e Virtualization T3 >80e Storage PaaS for Web Mutu, CloudWeb Compute Standalone, Cluster Game Collaborative Tools, NextCloud Database T4 >300e Bigdata T5 >600e HCI Plesk, CPanel AI PaaS with Platform.sh VDI Cloud Game Public Cloud 12KVA /32KVA Hosted Private Cloud K8S, IA IaaS PaaS for DevOps Storage File, Block, Object, Archive Databases SQL, noSQL, Messaging, Dashboard Network Virtual servers VPS, Dedicated Server Network VPS aaS pCC DC SaaS CRM, Billing, Payment, Stats IP FO, NAT, LB, VPN, Router, DNS, DHCP, TCP/SSL Offload Virtuozzo Cloud Security Wordpress, Magento, Prestashop Wholesales Hosted Private Cloud IAM, MFA, Encrypt, KMS IT Integrators, Cloud Storage, VMware SDDC, vSAN 1AZ / 2AZ vCD, Tanzu, Horizon, DBaaS, DRaaS Nutanix HCI 1AZ / 2AZ, Databases, DRaaS, VDI OpenStack IAM, Compute (VM, K8S) Stortage, Network, Databases Storage Ontap Select, Nutanix File OpenIO, MinIO, CEPH Zerto, Veeam, Atempo AI ElementAI, HuggingFace, Deepopmatic, Systran, EarthCube Bigdata / Analitics / ML Cloudera over S3, Dataiku, Saagie, Tableau, MarketPlace CDN, Database, ISV, WebHosting Support, Managed High Intensive CPU/GPU, Support Basic Encrypt Support thought Partners KMS, HSM Managed services Encrypt (SGX, Network, Storage) IA, DL Hybrid Cloud Standard Tools for AI, AI Studio, vRack Connect, Edge-DC, Private DC IA IaaS, Hosting API AI Dell, HP, Cisco, OCP, MultiCloud Bigdata, ML, Analytics Datalake, ML, Dashboard Secured Cloud GOV, FinTech, Retail, HealtCare
DevOps and Agility at OVHcloud You build it step by step, and you run it
A small fish in a big pond Needs to be adaptable, flexible, quick
Huge range of products For a comparatively small technical staff
With a wide variety of technology stacks Open source, non open source and self built
Only a way to keep up You build it iteratively, you run it
An Agile DevOps approach was natural Even if not always fully structured
Why do you mix DevOps and Agility? You can have one without the other, can’t you?
Agile is a response to Waterfall Methodological AND philosophical response
You don’t build an app like a bridge Static vs evolving paradigms
IT is an ever-evolving environment Agility is about speed of embracing change
You could have Agility without DevOps Even if I fail to see why you would want…
DevOps is a reaction to the wall of confusion Making the different stakeholders to work together in sync
You cannot have DevOps without agility DevOps is about shorter development sprints, increased focus on testing, increasing automation
DevOps comes with Agility DevOps is an extension of Agile that includes systems and operations
Let’s create a new product OVHcloud Managed Kubernetes
In 2018 we decided to build a product OVHcloud Managed Kubernetes
Build over our Public Cloud We leverage on our OpenStack expertise
Building a team for the project A small DevOps team to: ● Bootstrap the project ● Build the product ● Operate it
A balanced mix of skill needed At the beginning of the project: ● Architects ● Developers ● Sysadmins/Ops
But a clear objective By the end of the projects, having DevOps
Everybody on call The most direct road from Dev to DevOps
Developers need to be on call ● Engineering is about building and maintaining services ● On call should not be life-impacting ● Services are better when feedback loops are short
On-call duty needs to be humane ● Paid on-call duty ○ Flat rate per rotation period ○ Call out fee for responding to an alert. ● Enough people in rotation ○ Between 5-10 people ● Flexibility ○ Allowing people to swap on call time ● Curated alerts ○ Not all systems or errors deserve an on-call alert ● A clear escalation policy ○ And level 2/3 on-call duties
Building it for efficiency Because we need to facilitate operations
Kubinception: running K8s on K8s
And the ETCD
And the agility in all that? The keystone of the whole project and DevOps approach
DevOps teams all around the country ● ● ● ● ● ● ● ● Roubaix Paris Rennes Bordeaux Toulouse Lyon Nantes Brest
Tools & Methodology
What do we mean by agility?
A team with prejudices about Agility
st 1 step: building trust “I’m here to give you visibility” “We will create value together” “Your rhythm will be respected” “We will keep our commitments”
nd 2 step: team engagements “We respect the rules of Scrum” “We track our Impediments” “We respect our tickets’ lifecycle”
3rd step: convince with numbers and metrics!
Agility and real life of a DevOps team “I have a customer who came to ask me to solve a problem” “I am too often disturbed to fix bugs!” “I didn’t have time to finish the ticket, I had to deal with an emergency”
Impediments Unexpected tasks asking for immediate attention and diverting time from scheduled tasks
Tracking burndown and impediments
✔ Decision making ✔ Vision ✔ Creation of value ✔ Respect for the rhythm of the team
th 4 step: agile telemetry
Agile telemetry workflow
Extracting metrics from Jira
From metrics to visualisation
And then to the dashboard
OVHcloud agile telemetry MACRO • • • MICRO • • •
Last but not least… “We have kept our promises, with a fairly good level of quality” - Developer “We don’t apply the method mindlessly, we adapt it” Developer “Product release with good visibility and accuracy” - Manager A convinced product team
That’s all, folks! Thank you all!
Several years ago, at OVHcloud, we aligned our organisation to work in an Agile DevOps paradigm. We begin the talk explaining why we have done it, and how it’s a very pragmatical movement, not a philosophical one.
In the presentation we will also discuss what we understand for Agility (spoiler, it doesn’t forcible means Scrum or Kanban, it’s more a a way of work than a set of rules engraved on stone), what DevOps means for us, and why they are tightly linked.
We illustrate all that with a very concret example: the OVHcloud Managed Kubernetes team. We tell you the genesis of the project and the team, and how the Agile DevOps paradigm was implemented in the project, even with a team who didn’t forcibly believed it could work.
