A presentation at Codemotion Webinar by Horacio Gonzalez
Kubernetes Operators An introduction Horacio Gonzalez - @LostInBrittany October 27th, 2020
Who are we? Introducing myself and introducing OVH OVHcloud
Horacio Gonzalez @LostInBrittany Spaniard lost in Brittany, developer, dreamer and all-around geek Flutter
OVHcloud: A Global Leader 200k Private cloud VMs running 1 Dedicated IaaS Europe 30 Datacenters Own 20Tbps Hosting capacity : 1.3M Physical Servers 360k Servers already deployed Netwok with 35 PoPs
1.3M Customers in 138 Countries
OVHcloud: 4 Universes of Products WebCloud Domain / Email Domain names, DNS, SSL, Redirect Email, Open-Xchange, Exchange Baremetal Cloud VM General Purpose Baremetal SuperPlan T2 >20e Virtualization T3 >80e Storage PaaS for Web Mutu, CloudWeb Compute Standalone, Cluster Game Collaborative Tools, NextCloud Database T4 >300e Bigdata T5 >600e HCI Plesk, CPanel AI PaaS with Platform.sh VDI Cloud Game Public Cloud 12KVA /32KVA Hosted Private Cloud K8S, IA IaaS PaaS for DevOps Storage File, Block, Object, Archive Databases SQL, noSQL, Messaging, Dashboard Network Virtual servers VPS, Dedicated Server Network VPS aaS pCC DC SaaS CRM, Billing, Payment, Stats IP FO, NAT, LB, VPN, Router, DNS, DHCP, TCP/SSL Offload Virtuozzo Cloud Security Wordpress, Magento, Prestashop Wholesales Hosted Private Cloud IAM, MFA, Encrypt, KMS IT Integrators, Cloud Storage, VMware SDDC, vSAN 1AZ / 2AZ vCD, Tanzu, Horizon, DBaaS, DRaaS Nutanix HCI 1AZ / 2AZ, Databases, DRaaS, VDI OpenStack IAM, Compute (VM, K8S) Stortage, Network, Databases Storage Ontap Select, Nutanix File OpenIO, MinIO, CEPH Zerto, Veeam, Atempo AI ElementAI, HuggingFace, Deepopmatic, Systran, EarthCube Bigdata / Analitics / ML Cloudera over S3, Dataiku, Saagie, Tableau, MarketPlace CDN, Database, ISV, WebHosting Support, Managed High Intensive CPU/GPU, Support Basic Encrypt Support thought Partners KMS, HSM Managed services Encrypt (SGX, Network, Storage) IA, DL Hybrid Cloud Standard Tools for AI, AI Studio, vRack Connect, Edge-DC, Private DC IA IaaS, Hosting API AI Dell, HP, Cisco, OCP, MultiCloud Bigdata, ML, Analytics Datalake, ML, Dashboard Secured Cloud GOV, FinTech, Retail, HealtCare
OVHcloud Managed Kubernetes You use it, we operate it
Built over our Openstack based Public Cloud
Some interesting features
Operating Kubernetes Easier said than done
Operating microservices? Are you sure you want to operate them by hand?
Taming microservices with Kubernetes
Declarative infrastructure
Desired State Management
Beyond a simple deployment Everything is good now, isn’t it?
Complex deployments
Complex deployments
Helm Charts are configuration Operating is more than installs & upgrades
Kubernetes is about automation How about automating human operators?
Kubernetes Controllers Keeping an eye on the resources
A control loop They watch the state of the cluster, and make or request changes where needed
A reconcile loop Strives to reconcile current state and desired state
Custom Resource Definitions Extending Kubernetes API
Extending Kubernetes API By defining new types of resources
Kubernetes Operator Automating operations
What’s a Kubernetes Operator?
Example: databases Things like adding an instance to a pool, doing a backup, sharding…
Knowledge encoded in CRDs and Controllers
Custom Controllers for Custom Resources Operators implement and manage Custom Resources using custom reconciliation logic
Operator Capability Model Gauging the operator maturity
How to write an Operator
Kubebuilder SDK for building Kubernetes APIs using CRDs
The Operator Framework Open source framework to accelerate the development of an Operator
Operator SDK Three different ways to build an Operator
Operator SDK and Capability Model
Operator Lifecycle Manager
OperatorHub.io
Harbor Operator Managing private registries at scale
We wanted to build a new product OVHcloud Managed Private Registry
Looking at the Open Source world Two main alternatives around Docker Registry
Harbor has more community traction Two main alternatives
Harbor has lots of components
But it has a Helm Chart It should be easy to install, isn’t it? $ helm install harbor What about configuration? Installing a 200 GB K8s volume? Nginx pods for routing requests? One DB instance per customer? Managing pods all around the cluster?
We wanted a Managed Private Registry
Using the platform Kubernetes tooling to the rescue
Let’s automate it We needed an operator… and there wasn’t any
Working with the community Harbor community also needed the operator
The challenge: reconciliation loop
The Harbor Operator
It’s Open Source https://github.com/goharbor/harbor-operator
LoadBalancer Operator A managed LoadBalancer at scale
Load Balancer: a critical cog Cornerstone of any Cloud Provider’s infrastructure
Our legacy Load Balancer stack ● Excellent performances ○ ○ Built on bare metal servers + BGP Custom made servers tuned for network traffic ● Carry the TLS termination ○ SSL / LetsEncrypt ● Not cloud ready ○ ○ Piloted by configuration files Long configuration loading time ● Custom made hardware ○ ○ Slower to build Needs to be deployed on 30 datacenters
Our needs for a new Load Balancer ● Supporting mass update ● Quickly reconfigurable ● Available anywhere quickly ● Easily operable ● Integrated into our Public Cloud
Building it on Kubernetes
A Load Balancer in a pod
Orchestrating one million LBs… kubectl apply -f lb is not an option!
We needed an Operator
Network: multus-cni Attaching multiple network interfaces to pods: Bridge + Host-local
Adding network interfaces on the fly Using annotations to add interfaces to pod
Config management Using Config Map How to detect a change on Config Map files? Watch + Trigger? More information on Config Map working martensson.io/go-fsnotify-and-kubernetes-configmaps
A Controller to watch and trigger
Observability Tried Prometheus Operator, limited to one container per pod Switched to Warp 10 with Beamium Operator
That’s all, folks! Thank you all!
Operators are extensions to Kubernetes that simplify application install and management by leveraging on manage applications Custom Resources.
The Kubernetes Operator pattern tries the emulate the role of an human operator, who uses their deep knowledge of the application to install, operate and debug it. The Kubernetes Operators search to automate these tasks and facilitate the whole application life-cycle.
In this talk, we will explain how do we use Kubernetes Operators at OVHcloud, and how the help us to operate our Managed Kubernetes service at scale.
We will illustrate the talks with three concrete examples: Harbor Operator, LoadBalancing Operator and our incoming NodePool operator.
