Kubernetes Operators: Operating Cloud Native services at scale

Kubernetes Operators: Operating Cloud Native services at scale Horacio Gonzalez 2021-09-10

Who are we? Introducing myself and introducing OVH OVHcloud

Horacio Gonzalez @LostInBrittany Spaniard lost in Brittany, developer, dreamer and all-around geek Flutter

OVHcloud: A global leader Web Cloud & Telcom 30 Data Centers in 12 locations 1 Million+ Servers produced since 1999 Private Cloud 34 Points of Presence on a 20 TBPS Bandwidth Network 1.5 Million Customers across 132 countries Public Cloud 2200 Employees worldwide 3.8 Million Websites hosting Storage 115K Private Cloud VMS running 1.5 Billion Euros Invested since 2016 300K Public Cloud instances running P.U.E. 1.09 Energy efficiency indicator 380K Physical Servers running in our data centers 20+ Years in Business Disrupting since 1999 Network & Security

High performance at affordable prices From bare-metal servers to public or private cloud

Kubernetes Operators Helping to tame the complexity of K8s Ops

Taming microservices with Kubernetes

What about complex deployments

Specially at scale Lots of clusters with lots and lots of deployments

That’s just our case We both use Kubernetes and operate a Managed Kubernetes platform

Built over our Openstack based Public Cloud

We need to tame the complexity

Taming the complexity

Helm Charts are configuration Operating is more than installs & upgrades

Kubernetes is about automation How about automating human operators?

Kubernetes Operators A Kubernetes version of the human operator

Building operators Basic K8s elements: Controllers and Custom Resources

Kubernetes Controllers Keeping an eye on the resources

A control loop They watch the state of the cluster, and make or request changes where needed

A reconcile loop Strives to reconcile current state and desired state

Custom Resource Definitions Extending Kubernetes API

Extending Kubernetes API By defining new types of resources

Kubernetes Operator Automating operations

What’s a Kubernetes Operator?

Example: databases Things like adding an instance to a pool, doing a backup, sharding…

Knowledge encoded in CRDs and Controllers

Custom Controllers for Custom Resources Operators implement and manage Custom Resources using custom reconciliation logic

Operator Capability Model Gauging the operator maturity

How to write an Operator

Kubebuilder SDK for building Kubernetes APIs using CRDs

The Operator Framework Open source framework to accelerate the development of an Operator

Operator SDK Three different ways to build an Operator

Operator SDK and Capability Model

Operator Lifecycle Manager

OperatorHub.io

Harbor Operator Managing private registries at scale

We wanted to build a new product OVHcloud Managed Private Registry

Looking at the Open Source world Two main alternatives around Docker Registry

Harbor has more community traction Two main alternatives

Harbor has lots of components

But it has a Helm Chart It should be easy to install, isn’t it? $ helm install harbor What about configuration? Installing a 200 GB K8s volume? Nginx pods for routing requests? One DB instance per customer? Managing pods all around the cluster?

We wanted a Managed Private Registry

Using the platform Kubernetes tooling to the rescue

Let’s automate it We needed an operator… and there wasn’t any

Working with the community Harbor community also needed the operator

The challenge: reconciliation loop

The Harbor Operator

It’s Open Source https://github.com/goharbor/harbor-operator

LoadBalancer Operator A managed LoadBalancer at scale

Load Balancer: a critical cog Cornerstone of any Cloud Provider’s infrastructure

Our legacy Load Balancer stack ● Excellent performances ○ Built on bare metal servers + BGP ○ Custom made servers tuned for network traffic ● Carry the TLS termination ○ SSL / LetsEncrypt ● Not cloud ready ○ Piloted by configuration files ○ Long configuration loading time ● Custom made hardware ○ Slower to build ○ Needs to be deployed on 30 datacenters

Our needs for a new Load Balancer ● Supporting mass update ● Quickly reconfigurable ● Available anywhere quickly ● Easily operable ● Integrated into our Public Cloud

Building it on Kubernetes

A Load Balancer in a pod

Orchestrating one million LBs… kubectl apply -f lb is not an option!

We needed an Operator

Network: multus-cni Attaching multiple network interfaces to pods: Bridge + Host-local

Adding network interfaces on the fly Using annotations to add interfaces to pod

Config management Using Config Map How to detect a change on Config Map files? Watch + Trigger? More information on Config Map working martensson.io/go-fsnotify-and-kubernetes-configmaps

A Controller to watch and trigger

Observability Tried Prometheus Operator, limited to one container per pod Switched to Warp 10 with Beamium Operator

That’s all, folks! Thank you all!