OVHcloud K8s Team: DevOps & Agility in a Product Team

OVHcloud K8s Team DevOps & Agility in a Product Team Horacio Gonzalez 2020-07-09

Who are we? Introducing myself and introducing OVH OVHcloud

Horacio Gonzalez @LostInBrittany Spaniard lost in Brittany, developer, dreamer and all-around geek Flutter

OVHcloud: A Global Leader 200k Private cloud VMs running 1 Dedicated IaaS Europe 30 Datacenters Own 20Tbps Hosting capacity : 1.3M Physical Servers 360k Servers already deployed Netwok with 35 PoPs

1.3M Customers in 138 Countries

OVHcloud: 4 Universes of Products WebCloud Domain / Email Domain names, DNS, SSL, Redirect Email, Open-Xchange, Exchange Baremetal Cloud VM General Purpose Baremetal SuperPlan T2 >20e Virtualization T3 >80e Storage PaaS for Web Mutu, CloudWeb Compute Standalone, Cluster Game Collaborative Tools, NextCloud Database T4 >300e Bigdata T5 >600e HCI Plesk, CPanel AI PaaS with Platform.sh VDI Cloud Game Public Cloud 12KVA /32KVA Hosted Private Cloud K8S, IA IaaS PaaS for DevOps Storage File, Block, Object, Archive Databases SQL, noSQL, Messaging, Dashboard Network Virtual servers VPS, Dedicated Server Network VPS aaS pCC DC SaaS CRM, Billing, Payment, Stats IP FO, NAT, LB, VPN, Router, DNS, DHCP, TCP/SSL Offload Virtuozzo Cloud Security Wordpress, Magento, Prestashop Wholesales Hosted Private Cloud IAM, MFA, Encrypt, KMS IT Integrators, Cloud Storage, VMware SDDC, vSAN 1AZ / 2AZ vCD, Tanzu, Horizon, DBaaS, DRaaS Nutanix HCI 1AZ / 2AZ, Databases, DRaaS, VDI OpenStack IAM, Compute (VM, K8S) Stortage, Network, Databases Storage Ontap Select, Nutanix File OpenIO, MinIO, CEPH Zerto, Veeam, Atempo AI ElementAI, HuggingFace, Deepopmatic, Systran, EarthCube Bigdata / Analitics / ML Cloudera over S3, Dataiku, Saagie, Tableau, MarketPlace CDN, Database, ISV, WebHosting Support, Managed High Intensive CPU/GPU, Support Basic Encrypt Support thought Partners KMS, HSM Managed services Encrypt (SGX, Network, Storage) IA, DL Hybrid Cloud Standard Tools for AI, AI Studio, vRack Connect, Edge-DC, Private DC IA IaaS, Hosting API AI Dell, HP, Cisco, OCP, MultiCloud Bigdata, ML, Analytics Datalake, ML, Dashboard Secured Cloud GOV, FinTech, Retail, HealtCare

DevOps and Agility at OVHcloud You build it step by step, and you run it

A small fish in a big pond Needs to be adaptable, flexible, quick

Huge range of products For a comparatively small technical staff

With a wide variety of technology stacks Open source, non open source and self built

Only a way to keep up You build it iteratively, you run it

An Agile DevOps approach was natural Even if not always fully structured

Why do you mix DevOps and Agility? You can have one without the other, can’t you?

Agile is a response to Waterfall Methodological AND philosophical response

You don’t build an app like a bridge Static vs evolving paradigms

IT is an ever-evolving environment Agility is about speed of embracing change

You could have Agility without DevOps Even if I fail to see why you would want…

DevOps is a reaction to the wall of confusion Making the different stakeholders to work together in sync

You cannot have DevOps without agility DevOps is about shorter development sprints, increased focus on testing, increasing automation

DevOps comes with Agility DevOps is an extension of Agile that includes systems and operations

Let’s create a new product OVHcloud Managed Kubernetes

In 2018 we decided to build a product OVHcloud Managed Kubernetes

Build over our Public Cloud We leverage on our OpenStack expertise

Building a team for the project A small DevOps team to: ● Bootstrap the project ● Build the product ● Operate it

A balanced mix of skill needed At the beginning of the project: ● Architects ● Developers ● Sysadmins/Ops

But a clear objective By the end of the projects, having DevOps

Everybody on call The most direct road from Dev to DevOps

Developers need to be on call ● Engineering is about building and maintaining services ● On call should not be life-impacting ● Services are better when feedback loops are short

On-call duty needs to be humane ● Paid on-call duty ○ Flat rate per rotation period ○ Call out fee for responding to an alert. ● Enough people in rotation ○ Between 5-10 people ● Flexibility ○ Allowing people to swap on call time ● Curated alerts ○ Not all systems or errors deserve an on-call alert ● A clear escalation policy ○ And level 2/3 on-call duties

Building it for efficiency Because we need to facilitate operations

Kubinception: running K8s on K8s

And the ETCD

And the agility in all that? The keystone of the whole project and DevOps approach

DevOps teams all around the country ● ● ● ● ● ● ● ● Roubaix Paris Rennes Bordeaux Toulouse Lyon Nantes Brest

Tools & Methodology

What do we mean by agility?

A team with prejudices about Agility

st 1 step: building trust “I’m here to give you visibility” “We will create value together” “Your rhythm will be respected” “We will keep our commitments”

nd 2 step: team engagements “We respect the rules of Scrum” “We track our Impediments” “We respect our tickets’ lifecycle”

3rd step: convince with numbers and metrics!

Agility and real life of a DevOps team “I have a customer who came to ask me to solve a problem” “I am too often disturbed to fix bugs!” “I didn’t have time to finish the ticket, I had to deal with an emergency”

Impediments Unexpected tasks asking for immediate attention and diverting time from scheduled tasks

Tracking burndown and impediments

✔ Decision making ✔ Vision ✔ Creation of value ✔ Respect for the rhythm of the team

th 4 step: agile telemetry

Agile telemetry workflow

Extracting metrics from Jira

From metrics to visualisation

And then to the dashboard

OVHcloud agile telemetry MACRO • • • MICRO • • •

Last but not least… “We have kept our promises, with a fairly good level of quality” - Developer “We don’t apply the method mindlessly, we adapt it” Developer “Product release with good visibility and accuracy” - Manager A convinced product team

That’s all, folks! Thank you all!