From Minikube to Production Never Miss a Step in Getting Your K8s Ready Kevin Georges Horacio Gonzalez
A presentation at Kubecon Europe 2020 in August 2020 in by Horacio Gonzalez
From Minikube to Production Never Miss a Step in Getting Your K8s Ready Kevin Georges Horacio Gonzalez
Who are we? Introducing ourselves and introducing OVHcloud
Horacio Gonzalez @LostInBrittany Spaniard lost in Brittany, developer, dreamer and all-around geek Flutter
Kevin Georges @0xd33d33 Kubernetes Engineering Manager
OVHcloud: A Global Leader 200k Private cloud VMs running Dedicated 1 IaaS Europe Hosting capacity : 1.3M Physical Servers 360k Servers already deployed 30 Datacenters Own 20Tbps Netwok with 35 PoPs
1.3M Customers in 138 Countries
OVHcloud: 4 Universes of Products WebCloud Domain / Email Baremetal Cloud Compute Standalone, Cluster VM Domain names, DNS, SSL, Redirect General Purpose Email, Open-Xchange, Exchange Collaborative Tools, NextCloud Baremetal SuperPlan Game T2 >20e Virtualization Mutu, CloudWeb Plesk, CPanel Database T4 >300e Bigdata T5 >600e HCI AI 12KVA /32KVA VDI Cloud Game PaaS with Platform.sh Network VPS aaS MarketPlace Storage File, Block, Object, Archive VMware SDDC, vSAN 1AZ / 2AZ vCD, Tanzu, Horizon, DBaaS, DRaaS Nutanix HCI 1AZ / 2AZ, Databases, DRaaS, VDI Databases SQL, noSQL, Messaging, Dashboard IP FO, NAT, LB, VPN, Router, DNS, DHCP, TCP/SSL Offload Virtuozzo Cloud Security Wordpress, Magento, Prestashop CRM, Billing, Payment, Stats PaaS for DevOps Network pCC DC SaaS K8S, IA IaaS OpenStack IAM, Compute (VM, K8S) Stortage, Network, Databases Storage Ontap Select, Nutanix File Virtual servers VPS, Dedicated Server Hosted Private Cloud Hosted Private Cloud T3 >80e Storage PaaS for Web Public Cloud Wholesales IAM, MFA, Encrypt, KMS Support, Managed High Intensive CPU/GPU, Support Basic Encrypt Support thought Partners KMS, HSM Managed services Encrypt (SGX, Network, Storage) AI ElementAI, HuggingFace, Deepopmatic, Systran, EarthCube Bigdata / Analitics / ML Cloudera over S3, Dataiku, Saagie, Tableau, IT Integrators, Cloud Storage, CDN, Database, ISV, WebHosting OpenIO, MinIO, CEPH Zerto, Veeam, Atempo IA, DL Hybrid Cloud Standard Tools for AI, AI Studio, vRack Connect, Edge-DC, Private DC IA IaaS, Hosting API AI Dell, HP, Cisco, OCP, MultiCloud Bigdata, ML, Analytics Datalake, ML, Dashboard Secured Cloud GOV, FinTech, Retail, HealtCare
Orchestrating containers Like herding cats… but in hard mode!
From bare metal to containers Another paradigm shift
Containers are easy… For developers
Less simple if you must operate them Like in a production context
And what about microservices? Are you sure you want to operate them by hand?
Taming microservices with Kubernetes
Desired State Management
Having identical, software defined environments
I have deployed on Minikube, woah! A great fastlane into Kubernetes
Running a full K8s in your laptop A great learning tool
Your laptop isn’t a true cluster Don’t expect real performances
Beyond the first deployment So I have deployed my distributed architecture on K8s, everything is good now, isn’t it?
Minikube is only the beginning
From Minikube to prod A journey not for the faint of heart
Kubernetes can be wonderful For both developers and devops
But it comes with a price…
An example among many others
An example among many others
An example among many others
An example among many others
An example among many others
The truth is somewhere inside…
A network example: KubeProxy KubeProxy: 3 proxy modes ● Userspace ● IPTables ● IPVS
A network example: KubeProxy iptables by default
A network example: KubeProxy
A network example: KubeProxy Cluster networking will be slower and slower
A network example: KubeProxy IPVS to the rescue!
Kubernetes networking is complex…
The storage dilemma
The storage dilemma Volumes are handle through CSI CSI provide an interface between Kubernetes and storage technologie
The storage dilemma Most CSI assume perfect sync between Kubernetes and the storage backend
The storage dilemma Storage backend are subject to errors or maintenance Potential state shifts between storage and Kubernetes
The storage dilemma I0724 13:03:20.853645 1 csi_handler.go:100] Error processing “csi-afcb533080943”: failed to attach: rpc error: code = NotFound desc = ControllerPublishVolume Volume not found
The storage dilemma I0724 13:03:19.012008 1 csi_handler.go:100] Error processing “csi-2259b290c”: failed to attach: rpc error: code = Internal desc = ControllerPublishVolume Attach Volume failed with error failed to attach 9aa1b78d-503d-49ec-8e51-11e7c7a2dee7 volume to ea295f86-9fa8-497a-aeb9-4ad27a99a8ce compute: Bad request with: [POST https://compute.cloud.net/v2.1/327b346ae2034427b84dd/servers/ea295f86-9fa8-4 97a-aeb9-4ad27a99b76de/os-volume_attachments], error message: {“badRequest”: {“message”: “Invalid input received: Invalid volume: Volume status must be available to reserve. (HTTP 400) (Request-ID: req-8c41d48a-9a32-4225-b423-8e84131a3db8)”, “code”: 400}}
The storage dilemma I0724 13:03:15.997499 1 csi_handler.go:100] Error processing “csi-69164e184900”: failed to attach: rpc error: code = Internal desc = ControllerPublishVolume Attach Volume failed with error disk 57dbca1b-9611-4496-a960-ab13e355g23a is attached to a different instance (1621db21-b4af-4bd8-9419-954ed70gh892)
The ETCD vulnerability
Security Hardening your Kubernetes
The security journey
Kubernetes is insecure by design* It’s a feature, not a bug. Up to K8s admin to secure it according to needs
Not everybody has the same security needs
Kubernetes allows to enforce security practices as needed
Listing some good practices
Security defaults Kubernetes is insecure by default:
Security defaults Kubernetes is insecure by default:
Close open access Close all by default, open only the needed ports Follow the least privileged principle
Define and implement RBAC According to your needs
Define and implement network policies
Use RBAC and Network Policies to isolate your sensitive workload
Always keep up to date Both Kubernetes and plugins
Because Kubernetes is a big target
And remember, even the best can get hacked Remain attentive, don’t get too confident
Extensibility Enhance your Kubernetes
Kubernetes is modular Let’s see how some of those plugins can help you
Helm A package management for K8s
Complex deployments
Using static YAML files
Complex deployments
Istio A service mesh for Kubernetes… and much more!
Istio: A service mesh… but not only
Service discovery
Traffic control
Encrypting internal communications
Routing and load balancing
Rolling upgrades
Rolling upgrades
A/B testing
Monitoring your cluster
Velero Backing up your Kubernetes
Kubernetes: Desired State Management
YAML files allows to clone a cluster
But what about the data?
Velero Backup and migrate Kubernetes applications and their persistent volumes
S3 based backup On any S3 protocol compatible store
Backup all or part of a cluster
Schedule backups
Backups hooks
Conclusion And one more thing…
Kubernetes is easy to begin with Minikube, K3s…
Kubernetes is powerful It can make Developers’ and DevOps’ lives easier
But there is a price: operating it Lot of things to think about
We have seen some of them
Different roles Each role asks for very different knowledge and skill sets
Operating a Kubernetes cluster is hard But we have a good news…
Most companies don’t need to do it! As they don’t build and rack their own servers!
If you don’t need to build it, choose a certified managed solution You get the cluster, the operator get the problems
Like our OVH Managed Kubernetes Made with 💗 by the Platform team
Do you want to try? Come to our (virtual) booth!
Thank you for listening
