Kubernetes in production: avoiding the pitfalls Horacio Gonzalez

Who are we? Introducing ourselves and introducing OVHcloud

Horacio Gonzalez @LostInBrittany Spaniard lost in Brittany, developer, dreamer and all-around geek Flutter

OVHcloud: A Global Leader 200k Private cloud VMs running Dedicated 1 IaaS Europe Hosting capacity : 1.3M Physical Servers 360k Servers already deployed 30 Datacenters Own 20Tbps Netwok with 35 PoPs

1.4M Customers in 138 Countries

OVHcloud: 4 Universes of Products WebCloud Domain / Email Baremetal Cloud Compute Standalone, Cluster VM Domain names, DNS, SSL, Redirect General Purpose Email, Open-Xchange, Exchange Collaborative Tools, NextCloud Baremetal SuperPlan Game T2 >20e Virtualization Mutu, CloudWeb Plesk, CPanel Database T4 >300e Bigdata T5 >600e HCI AI 12KVA /32KVA VDI Cloud Game PaaS with Platform.sh Network VPS aaS MarketPlace Storage File, Block, Object, Archive VMware SDDC, vSAN 1AZ / 2AZ vCD, Tanzu, Horizon, DBaaS, DRaaS Nutanix HCI 1AZ / 2AZ, Databases, DRaaS, VDI Databases SQL, noSQL, Messaging, Dashboard IP FO, NAT, LB, VPN, Router, DNS, DHCP, TCP/SSL Offload Virtuozzo Cloud Security Wordpress, Magento, Prestashop CRM, Billing, Payment, Stats PaaS for DevOps Network pCC DC SaaS K8S, IA IaaS OpenStack IAM, Compute (VM, K8S) Stortage, Network, Databases Storage Ontap Select, Nutanix File Virtual servers VPS, Dedicated Server Hosted Private Cloud Hosted Private Cloud T3 >80e Storage PaaS for Web Public Cloud Wholesales IAM, MFA, Encrypt, KMS Support, Managed High Intensive CPU/GPU, Support Basic Encrypt Support thought Partners KMS, HSM Managed services Encrypt (SGX, Network, Storage) AI ElementAI, HuggingFace, Deepopmatic, Systran, EarthCube Bigdata / Analitics / ML Cloudera over S3, Dataiku, Saagie, Tableau, IT Integrators, Cloud Storage, CDN, Database, ISV, WebHosting OpenIO, MinIO, CEPH Zerto, Veeam, Atempo IA, DL Hybrid Cloud Standard Tools for AI, AI Studio, vRack Connect, Edge-DC, Private DC IA IaaS, Hosting API AI Dell, HP, Cisco, OCP, MultiCloud Bigdata, ML, Analytics Datalake, ML, Dashboard Secured Cloud GOV, FinTech, Retail, HealtCare

Orchestrating containers Like herding cats… but in hard mode!

From bare metal to containers Another paradigm shift

Containers are easy… For developers

Less simple if you must operate them Like in a production context

And what about microservices? Are you sure you want to operate them by hand?

Taming microservices with Kubernetes

Desired State Management

Having identical, software defined environments

I have deployed on Minikube, woah! A great fastlane into Kubernetes

Running a full K8s in your laptop A great learning tool

Your laptop isn’t a true cluster Don’t expect real performances

Beyond the first deployment So I’ve deployed my distributed architecture on K8s, everything is good now, isn’t it?

Minikube is only the beginning

From Minikube to prod A journey not for the faint of heart

Kubernetes can be wonderful For both developers and devops

But it comes with a price…

An example among many others

An example among many others

An example among many others

An example among many others

An example among many others

The truth is somewhere inside…

A network example: KubeProxy KubeProxy: 3 proxy modes ● Userspace ● IPTables ● IPVS

A network example: KubeProxy iptables by default

A network example: KubeProxy

A network example: KubeProxy Cluster networking will be slower and slower

A network example: KubeProxy IPVS to the rescue!

Kubernetes networking is complex…

The storage dilemma

The storage dilemma Volumes are handle through CSI CSI provide an interface between Kubernetes and storage technologie

The storage dilemma Most CSI assume perfect sync between Kubernetes and the storage backend

The storage dilemma Storage backend are subject to errors or maintenance Potential state shifts between storage and Kubernetes

The ETCD vulnerability

Security Hardening your Kubernetes

The security journey

Kubernetes is insecure by design* It’s a feature, not a bug. Up to K8s admin to secure it according to needs

Not everybody has the same security needs

Kubernetes allows to enforce security practices as needed

Listing some good practices

Close open access Close all by default, open only the needed ports Follow the least privileged principle

Define and implement RBAC According to your needs

Define and implement network policies

Use RBAC and Network Policies to isolate your sensitive workload

Always keep up to date Both Kubernetes and plugins

Because Kubernetes is a big target

And remember, even the best can get hacked Remain attentive, don’t get too confident

Extensibility Enhance your Kubernetes

Kubernetes is modular Let’s see how some of those plugins can help you

Helm A package management for K8s

Complex deployments

Using static YAML files

Complex deployments

Istio A service mesh for Kubernetes… and much more!

Istio: A service mesh… but not only

Service discovery

Traffic control

Encrypting internal communications

Routing and load balancing

Rolling upgrades

Rolling upgrades

A/B testing

Monitoring your cluster

Velero Backing up your Kubernetes

Kubernetes: Desired State Management

YAML files allows to clone a cluster

But what about the data?

Velero Backup and migrate Kubernetes applications and their persistent volumes

S3 based backup On any S3 protocol compatible store

Backup all or part of a cluster

Schedule backups

Backups hooks

Conclusion And one more thing…

Kubernetes is easy to begin with Minikube, K3s…

Kubernetes is powerful It can make Developers’ and DevOps’ lives easier

But there is a price: operating it Lot of things to think about

We have seen some of them

Different roles Each role asks for very different knowledge and skill sets

Operating a Kubernetes cluster is hard But we have a good news…

Most companies don’t need to do it! As they don’t build and rack their own servers!

If you don’t need to build it, choose a certified managed solution You get the cluster, the operator get the problems

Like our OVH Managed Kubernetes Made with 💗 by the Platform team

Thank you for listening!