Kubernetes Operators: Operating Cloud Native services at scale

A presentation at Incontro DevOps Italia in July 2021 in by Horacio Gonzalez

Slide 1

Slide 1

Kubernetes Operators: Operating Cloud Native services at scale Horacio Gonzalez 2021-02-05

Slide 2

Slide 2

Who are we? Introducing myself and introducing OVH OVHcloud

Slide 3

Slide 3

Horacio Gonzalez @LostInBrittany Spaniard lost in Brittany, developer, dreamer and all-around geek Flutter

Slide 4

Slide 4

OVHcloud: A global leader Web Cloud & Telcom 30 Data Centers in 12 locations 1 Million+ Servers produced since 1999 Private Cloud 34 Points of Presence on a 20 TBPS Bandwidth Network 1.5 Million Customers across 132 countries Public Cloud 2200 Employees worldwide 3.8 Million Websites hosting Storage 115K Private Cloud VMS running 1.5 Billion Euros Invested since 2016 300K Public Cloud instances running P.U.E. 1.09 Energy efficiency indicator 380K Physical Servers running in our data centers 20+ Years in Business Disrupting since 1999 Network & Security

Slide 5

Slide 5

High performance at affordable prices From bare-metal servers to public or private cloud

Slide 6

Slide 6

Kubernetes Operators Helping to tame the complexity of K8s Ops

Slide 7

Slide 7

Taming microservices with Kubernetes

Slide 8

Slide 8

What about complex deployments

Slide 9

Slide 9

Specially at scale Lots of clusters with lots and lots of deployments

Slide 10

Slide 10

That’s just our case We both use Kubernetes and operate a Managed Kubernetes platform

Slide 11

Slide 11

Built over our Openstack based Public Cloud

Slide 12

Slide 12

We need to tame the complexity

Slide 13

Slide 13

Taming the complexity

Slide 14

Slide 14

Helm Charts are configuration Operating is more than installs & upgrades

Slide 15

Slide 15

Kubernetes is about automation How about automating human operators?

Slide 16

Slide 16

Kubernetes Operators A Kubernetes version of the human operator

Slide 17

Slide 17

Building operators Basic K8s elements: Controllers and Custom Resources

Slide 18

Slide 18

Kubernetes Controllers Keeping an eye on the resources

Slide 19

Slide 19

A control loop They watch the state of the cluster, and make or request changes where needed

Slide 20

Slide 20

A reconcile loop Strives to reconcile current state and desired state

Slide 21

Slide 21

Custom Resource Definitions Extending Kubernetes API

Slide 22

Slide 22

Extending Kubernetes API By defining new types of resources

Slide 23

Slide 23

Kubernetes Operator Automating operations

Slide 24

Slide 24

What’s a Kubernetes Operator?

Slide 25

Slide 25

Example: databases Things like adding an instance to a pool, doing a backup, sharding…

Slide 26

Slide 26

Knowledge encoded in CRDs and Controllers

Slide 27

Slide 27

Custom Controllers for Custom Resources Operators implement and manage Custom Resources using custom reconciliation logic

Slide 28

Slide 28

Operator Capability Model Gauging the operator maturity

Slide 29

Slide 29

How to write an Operator

Slide 30

Slide 30

Kubebuilder SDK for building Kubernetes APIs using CRDs

Slide 31

Slide 31

The Operator Framework Open source framework to accelerate the development of an Operator

Slide 32

Slide 32

Operator SDK Three different ways to build an Operator

Slide 33

Slide 33

Operator SDK and Capability Model

Slide 34

Slide 34

Operator Lifecycle Manager

Slide 35

Slide 35

OperatorHub.io

Slide 36

Slide 36

Harbor Operator Managing private registries at scale

Slide 37

Slide 37

We wanted to build a new product OVHcloud Managed Private Registry

Slide 38

Slide 38

Looking at the Open Source world Two main alternatives around Docker Registry

Slide 39

Slide 39

Harbor has more community traction Two main alternatives

Slide 40

Slide 40

Harbor has lots of components

Slide 41

Slide 41

But it has a Helm Chart It should be easy to install, isn’t it? $ helm install harbor What about configuration? Installing a 200 GB K8s volume? Nginx pods for routing requests? One DB instance per customer? Managing pods all around the cluster?

Slide 42

Slide 42

We wanted a Managed Private Registry

Slide 43

Slide 43

Using the platform Kubernetes tooling to the rescue

Slide 44

Slide 44

Let’s automate it We needed an operator… and there wasn’t any

Slide 45

Slide 45

Working with the community Harbor community also needed the operator

Slide 46

Slide 46

The challenge: reconciliation loop

Slide 47

Slide 47

The Harbor Operator

Slide 48

Slide 48

It’s Open Source https://github.com/goharbor/harbor-operator

Slide 49

Slide 49

LoadBalancer Operator A managed LoadBalancer at scale

Slide 50

Slide 50

Load Balancer: a critical cog Cornerstone of any Cloud Provider’s infrastructure

Slide 51

Slide 51

Our legacy Load Balancer stack ● Excellent performances ○ Built on bare metal servers + BGP ○ Custom made servers tuned for network traffic ● Carry the TLS termination ○ SSL / LetsEncrypt ● Not cloud ready ○ Piloted by configuration files ○ Long configuration loading time ● Custom made hardware ○ Slower to build ○ Needs to be deployed on 30 datacenters

Slide 52

Slide 52

Our needs for a new Load Balancer ● Supporting mass update ● Quickly reconfigurable ● Available anywhere quickly ● Easily operable ● Integrated into our Public Cloud

Slide 53

Slide 53

Building it on Kubernetes

Slide 54

Slide 54

A Load Balancer in a pod

Slide 55

Slide 55

Orchestrating one million LBs… kubectl apply -f lb is not an option!

Slide 56

Slide 56

We needed an Operator

Slide 57

Slide 57

Network: multus-cni Attaching multiple network interfaces to pods: Bridge + Host-local

Slide 58

Slide 58

Adding network interfaces on the fly Using annotations to add interfaces to pod

Slide 59

Slide 59

Config management Using Config Map How to detect a change on Config Map files? Watch + Trigger? More information on Config Map working martensson.io/go-fsnotify-and-kubernetes-configmaps

Slide 60

Slide 60

A Controller to watch and trigger

Slide 61

Slide 61

Observability Tried Prometheus Operator, limited to one container per pod Switched to Warp 10 with Beamium Operator

Slide 62

Slide 62

That’s all, folks! Thank you all!