A presentation at DevNexus 2020 in in Atlanta, GA, USA by Viktor Gamov
Kafka on Kubernetes: Does it have to be the hard way? february / Atlanta, GA 2019 @gamussa @gamussa || #devnexus #devnexus || @ConfluentINc @ConfluentINc
2 @gamussa | #devnexus | @ConfluentINc
3 I build highly scalable Hello World apps @gamussa | #devnexus | @ConfluentINc
5 Evolution of #devkafkaops Ansible Containers Cloud Scripts @gamussa | #devnexus | @ConfluentINc
7 @gamussa | #devnexus | @ConfluentINc
8 https://twitter.com/claesmogren/status/1108456688175079424 @gamussa | #devnexus | @ConfluentINc
9 https://twitter.com/claesmogren/status/1108456688175079424 https://twitter.com/QuinnyPig/status/1150927901782499330 @gamussa | #devnexus | @ConfluentINc
10 🙋 @gamussa | #devnexus | @ConfluentINc
11 Who run stateless apps in Kubernetes? 🙋 Who thinks it’s a good idea? Who run stateful apps in Kubernetes? Who thinks it’s a good idea? @gamussa | #devnexus | @ConfluentINc
12 #devkafkaops Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades @gamussa | #devnexus | @ConfluentINc
13 kafkaesque world of Kafka on Kubernetes @gamussa | #devnexus | @ConfluentINc
14 Dude, you said «it doesn’t have to be hard way» @gamussa | #devnexus | @ConfluentINc
DO KAFKA ON KUBERNETES DEMO AND EVERYONE LOOSES THEIR MINDS @gamussa | #devnexus | @ConfluentINc 15
Tweet, yeah 🚀 Follow @gamussa 📸🖼🏋😂 Tag @gamussa With #devnexus @confluentinc
17 Let’s talk Kubernetes @gamussa | #devnexus | @ConfluentINc
18 @gamussa | #devnexus | @ConfluentINc
19 Kubernetes ●Schedules and allocates resources ●Networking between Pods ●Storage ●Service Discovery @gamussa | #devnexus | @ConfluentINc
20 Kubernetes Architecture kubectl https://thenewstack.io/kubernetes-an-overview/ @gamussa | #devnexus | @ConfluentINc
21 @gamussa | #devnexus | @ConfluentINc
22 Pod • Basic Unit of Deployment in Kubernetes • A collection of containers sharing: • Namespace • Network • Volumes @gamussa | #devnexus | @ConfluentINc
23 Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV @gamussa | #devnexus | @ConfluentINc
24 Kubernetes Storage classes Fast Slow Admin @gamussa | #devnexus | @ConfluentINc
25 Kubernetes Storage classes Fast Slow Admin I feel the need for speed © Claim Developer @gamussa | #devnexus | @ConfluentINc
26 Kubernetes Storage classes Fast Slow Admin Fast PV I feel the need for speed © Pod Claim Developer @gamussa | #devnexus | @ConfluentINc
27 Kubernetes Storage classes Fast Slow Admin Fast PV I feel the need for speed © Pod Claim Claim Developer @gamussa | #devnexus | @ConfluentINc
28 Provisioning storage in Kubernetes Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc
29 Provisioning storage in Kubernetes The controller requests storage system to attach volume Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc
30 Provisioning storage in Kubernetes The storage system attaches the volume Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc
31 Provisioning storage in Kubernetes Kubelet requests storage system to mount Volume Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc
32 Provisioning storage in Kubernetes Kubelet can now request container runtime to bind mount the volume into the requested path in the container. Kubernetes controller Storage System Kubernetes node /var/lib @gamussa | #devnexus | @ConfluentINc
33 But I just want to deploy kafka @gamussa | #devnexus | @ConfluentINc
35 Confluent Operator: Apache Kafka on Kubernetes made simple Confluent Platform Docker Images Run Apache Kafka and Confluent Confluent Operator Platform as a cloud-native application on Kubernetes to minimize operating complexity and Kubernetes AWS Azure GCP RH OpenShift Mesosphere Pivotal increase developer agility On-Premises @gamussa | #devnexus | @ConfluentINc Cloud
Custom resource controller
37 a custom Kubernetes controller PODS API Server StatefulSets PVs Controllers & Custom Controllers Custom Resources Master Node Worker Node | Applications run on Kubernetes ● Applications use objects like StatefulSets, Configmaps, PVs ● Custom Controllers create custom resources that provide unique application functionality: ConfigMaps Scheduler @gamussa ● Nodes and pods are where #devnexus | @ConfluentINc ○ Upgrades, elasticity, Kafka Operational Logic
38 @gamussa | #devnexus | @ConfluentINc
39 Workloads Deployment @gamussa | #devnexus | @ConfluentINc
40 Usual suspects Certified container images @gamussa Operator Helm Charts | #devnexus | @ConfluentINc
41 We just need to deploy Kafka on Kubernetes @gamussa | #devnexus | @ConfluentINc
42 We will use confluentinc/cp-helmcharts @gamussa | #devnexus | @ConfluentINc
43 Helm Charts @gamussa | #devnexus | @ConfluentINc
44 Helm Charts @gamussa | #devnexus | @ConfluentINc
45 Helm Charts @gamussa | #devnexus | @ConfluentINc
46 https://cnfl.io/helm_video @gamussa | #devnexus | @ConfluentINc
47 Helm Charts is just a GO Templates. How Charts help with rolling update? @gamussa | #devnexus | @ConfluentINc
48 We will use StatefulSets with OrderedReady @gamussa | #devnexus | @ConfluentINc
49 We need SRE / Operator knowledge to manage the platform. You need Operator! @gamussa | #devnexus | @ConfluentINc
50 @gamussa | #devnexus | @ConfluentINc
Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under-replicated partitions 5. Upgrade the next broker
52 Will it Scale Spin up new brokers, connect workers easily Manual Rebalance required in v1.0 Determine balancing plan Execute balancing plan Monitor Resources @gamussa | #devnexus | @ConfluentINc
53 One More Thing…
54 Usual suspects Certified container images @gamussa | #devnexus | @ConfluentINc
55 @gamussa | #devnexus | @ConfluentINc
Thanks! @gamussa viktor@confluent.io https://slackpass.io/confluentcommunity #kubernetes @gamussa | @ #devnexus | @ConfluentINc
57
When it comes to choosing a distributed streaming platform for real-time data pipelines, everyone knows the answer: Apache Kafka! And when it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. However, with all great things, the devil is truly in the details. While Kubernetes does provide all the building blocks that are needed, a lot of thought is required to truly create an enterprise-grade Kafka platform that can be used in production. In this technical deep dive, Viktor will go through challenges and pitfalls of managing Kafka on Kubernetes as well as the goals and lessons learned from the development of the Confluent Operator for Kubernetes.
The following resources were mentioned during the presentation or are useful additional information.
twitter streams presentation
Here’s what was said about this presentation on social media.
