A presentation at JDev 2020 by Horacio Gonzalez
OVHCloud K8s DevOps & Agility in a Product Team Horacio Gonzalez 2020-07-09
Who are we? Introducing myself and introducing OVH OVHcloud
Horacio Gonzalez @LostInBrittany Spaniard lost in Brittany, developer, dreamer and all-around geek Flutter
OVHcloud: A Global Leader 200k Private cloud VMs running 1 Dedicated IaaS Europe 30 Datacenters Own 20Tbps Hosting capacity : 1.3M Physical Servers 360k Servers already deployed Netwok with 35 PoPs
1.3M Customers in 138 Countries
OVHcloud: 4 Universes of Products WebCloud Domain / Email Domain names, DNS, SSL, Redirect Email, Open-Xchange, Exchange Baremetal Cloud VM General Purpose Baremetal SuperPlan T2 >20e Virtualization T3 >80e Storage PaaS for Web Mutu, CloudWeb Compute Standalone, Cluster Game Collaborative Tools, NextCloud Database T4 >300e Bigdata T5 >600e HCI Plesk, CPanel AI PaaS with Platform.sh VDI Cloud Game Public Cloud 12KVA /32KVA Hosted Private Cloud K8S, IA IaaS PaaS for DevOps Storage File, Block, Object, Archive Databases SQL, noSQL, Messaging, Dashboard Network Virtual servers VPS, Dedicated Server Network VPS aaS pCC DC SaaS CRM, Billing, Payment, Stats IP FO, NAT, LB, VPN, Router, DNS, DHCP, TCP/SSL Offload Virtuozzo Cloud Security Wordpress, Magento, Prestashop Wholesales Hosted Private Cloud IAM, MFA, Encrypt, KMS IT Integrators, Cloud Storage, VMware SDDC, vSAN 1AZ / 2AZ vCD, Tanzu, Horizon, DBaaS, DRaaS Nutanix HCI 1AZ / 2AZ, Databases, DRaaS, VDI OpenStack IAM, Compute (VM, K8S) Stortage, Network, Databases Storage Ontap Select, Nutanix File OpenIO, MinIO, CEPH Zerto, Veeam, Atempo AI ElementAI, HuggingFace, Deepopmatic, Systran, EarthCube Bigdata / Analitics / ML Cloudera over S3, Dataiku, Saagie, Tableau, MarketPlace CDN, Database, ISV, WebHosting Support, Managed High Intensive CPU/GPU, Support Basic Encrypt Support thought Partners KMS, HSM Managed services Encrypt (SGX, Network, Storage) IA, DL Hybrid Cloud Standard Tools for AI, AI Studio, vRack Connect, Edge-DC, Private DC IA IaaS, Hosting API AI Dell, HP, Cisco, OCP, MultiCloud Bigdata, ML, Analytics Datalake, ML, Dashboard Secured Cloud GOV, FinTech, Retail, HealtCare
DevOps at OVHcloud You build it, you run it
A small fish in a big pond Needs to be adaptable, flexible, quick
Huge range of products For a comparatively small technical staff
With a wide variety of technology stacks Open source, non open source and self built
Only a way to keep up You build it, you run it
The DevOps approach was natural Even if not always fully structured
Let’s create a new product OVHcloud Managed Kubernetes
In 2018 we decided to build a product OVHcloud Managed Kubernetes
Build over our Public Cloud We leverage on our OpenStack expertise
Building a team for the project A small DevOps team to: ● Bootstrap the project ● Build the product ● Operate it
A balanced mix of skill needed At the beginning of the project: ● Architects ● Developers ● Sysadmins/Ops
But a clear objective By the end of the projects, having DevOps
Everybody on call The most direct road from Dev to DevOps
Developers need to be on call ● Engineering is about building and maintaining services ● On call should not be life-impacting ● Services are better when feedback loops are short
On-call duty needs to be humane ● Paid on-call duty ○ Flat rate per rotation period ○ Call out fee for responding to an alert. ● Enough people in rotation ○ Between 5-10 people ● Flexibility ○ Allowing people to swap on call time ● Curated alerts ○ Not all systems or errors deserve an on-call alert ● A clear escalation policy ○ And level 2/3 on-call duties
Building it for efficiency Because we need to facilitate operations
Kubinception: running K8s on K8s
And the ETCD
And the agility in all that? The keystone of the whole project and DevOps approach
DevOps teams all around the country ● ● ● ● ● ● ● ● Roubaix Paris Rennes Bordeaux Toulouse Lyon Nantes Brest
Tools & Methodology « SCRUM » « Farmer’s Wisdom » « KINTSUGI* »
What do we mean by agility?
A team with prejudices about Agility “Being Agile? This is a trick to justify a failure in project management. ” “The Scrum does not work for development teams, it is for managers” “Scrum is only useful to watch and control us” “Scrum? We tested 3 years ago, we quickly understood that it was not for us…”
st 1 step: building trust “I’m here to give you visibility” “We will create value together” “Your rhythm will be respected” “We will keep our commitments”
nd 2 step: team engagements “We respect the rules of Scrum” “We track our Impediments” “We respect our tickets’ lifecycle”
3rd step: convince with numbers and metrics!
Agility and real life of a DevOps team “I have a customer who came to ask me to solve a problem” “I am too often disturbed to fix bugs!” “I didn’t have time to finish the ticket, I had to deal with an emergency”
Impediments Unexpected tasks asking for immediate attention and diverting time from scheduled tasks
Tracking burndown and impediments
✔ Decision making ✔ Vision ✔ Creation of value ✔ Respect for the rhythm of the team
Agile telemetry workflow
Extracting metrics from Jira bot.jerem Language JIRA Metrics DataBase Golang API Scrapping HTTP POST Warp10
From metrics to visualisation 1 Objet 3 pts 3 pts 3 pts 5 pts 5 pts 8 pts 8 pts 8 pts Example : Projet : OB Epic : Alpha Information : taches DONE 12 pts 18 pts Temps Interval régulier envir : 20min
And then to the dashboard
OVHcloud agile telemetry MACRO • Top management • Product Marketing • Program Manager MICRO • Equipe de dev • Scrum Master • Technical leader
Last but not least… “We have kept our promises, with a fairly good level of quality” - Developer “We don’t apply the method mindlessly, we adapt it” Developer “Product release with good visibility and accuracy” - Manager A convinced product team
That’s all, folks! Thank you all!
The following resources were mentioned during the presentation or are useful additional information.
