Kafka on Kubernetes!

Kafka on Kubernetes: Does it have to be the hard way? february / Atlanta, GA 2019 @gamussa @gamussa || #devnexus #devnexus || @ConfluentINc @ConfluentINc

2 @gamussa | #devnexus | @ConfluentINc

3 I build highly scalable Hello World apps @gamussa | #devnexus | @ConfluentINc

5 Evolution of #devkafkaops Ansible Containers Cloud Scripts @gamussa | #devnexus | @ConfluentINc

7 @gamussa | #devnexus | @ConfluentINc

8 https://twitter.com/claesmogren/status/1108456688175079424 @gamussa | #devnexus | @ConfluentINc

9 https://twitter.com/claesmogren/status/1108456688175079424 https://twitter.com/QuinnyPig/status/1150927901782499330 @gamussa | #devnexus | @ConfluentINc

10 🙋 @gamussa | #devnexus | @ConfluentINc

11 Who run stateless apps in Kubernetes? 🙋 Who thinks it’s a good idea? Who run stateful apps in Kubernetes? Who thinks it’s a good idea? @gamussa | #devnexus | @ConfluentINc

12 #devkafkaops Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades @gamussa | #devnexus | @ConfluentINc

13 kafkaesque world of Kafka on Kubernetes @gamussa | #devnexus | @ConfluentINc

14 Dude, you said «it doesn’t have to be hard way» @gamussa | #devnexus | @ConfluentINc

DO KAFKA ON KUBERNETES DEMO AND EVERYONE LOOSES THEIR MINDS @gamussa | #devnexus | @ConfluentINc 15

Tweet, yeah 🚀 Follow @gamussa 📸🖼🏋😂 Tag @gamussa With #devnexus @confluentinc

17 Let’s talk Kubernetes @gamussa | #devnexus | @ConfluentINc

18 @gamussa | #devnexus | @ConfluentINc

19 Kubernetes ●Schedules and allocates resources ●Networking between Pods ●Storage ●Service Discovery @gamussa | #devnexus | @ConfluentINc

20 Kubernetes Architecture kubectl https://thenewstack.io/kubernetes-an-overview/ @gamussa | #devnexus | @ConfluentINc

21 @gamussa | #devnexus | @ConfluentINc

22 Pod • Basic Unit of Deployment in Kubernetes • A collection of containers sharing: • Namespace • Network • Volumes @gamussa | #devnexus | @ConfluentINc

23 Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV @gamussa | #devnexus | @ConfluentINc

24 Kubernetes Storage classes Fast Slow Admin @gamussa | #devnexus | @ConfluentINc

25 Kubernetes Storage classes Fast Slow Admin I feel the need for speed © Claim Developer @gamussa | #devnexus | @ConfluentINc

26 Kubernetes Storage classes Fast Slow Admin Fast PV I feel the need for speed © Pod Claim Developer @gamussa | #devnexus | @ConfluentINc

27 Kubernetes Storage classes Fast Slow Admin Fast PV I feel the need for speed © Pod Claim Claim Developer @gamussa | #devnexus | @ConfluentINc

28 Provisioning storage in Kubernetes Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc

29 Provisioning storage in Kubernetes The controller requests storage system to attach volume Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc

30 Provisioning storage in Kubernetes The storage system attaches the volume Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc

31 Provisioning storage in Kubernetes Kubelet requests storage system to mount Volume Kubernetes controller Storage System Kubernetes node @gamussa | #devnexus | @ConfluentINc

32 Provisioning storage in Kubernetes Kubelet can now request container runtime to bind mount the volume into the requested path in the container. Kubernetes controller Storage System Kubernetes node /var/lib @gamussa | #devnexus | @ConfluentINc

33 But I just want to deploy kafka @gamussa | #devnexus | @ConfluentINc

35 Confluent Operator: Apache Kafka on Kubernetes made simple Confluent Platform Docker Images Run Apache Kafka and Confluent Confluent Operator Platform as a cloud-native application on Kubernetes to minimize operating complexity and Kubernetes AWS Azure GCP RH OpenShift Mesosphere Pivotal increase developer agility On-Premises @gamussa | #devnexus | @ConfluentINc Cloud

Custom resource controller

37 a custom Kubernetes controller PODS API Server StatefulSets PVs Controllers & Custom Controllers Custom Resources Master Node Worker Node | Applications run on Kubernetes ● Applications use objects like StatefulSets, Configmaps, PVs ● Custom Controllers create custom resources that provide unique application functionality: ConfigMaps Scheduler @gamussa ● Nodes and pods are where #devnexus | @ConfluentINc ○ Upgrades, elasticity, Kafka Operational Logic

38 @gamussa | #devnexus | @ConfluentINc

39 Workloads Deployment @gamussa | #devnexus | @ConfluentINc

40 Usual suspects Certified container images @gamussa Operator Helm Charts | #devnexus | @ConfluentINc

41 We just need to deploy Kafka on Kubernetes @gamussa | #devnexus | @ConfluentINc

42 We will use confluentinc/cp-helmcharts @gamussa | #devnexus | @ConfluentINc

43 Helm Charts @gamussa | #devnexus | @ConfluentINc

44 Helm Charts @gamussa | #devnexus | @ConfluentINc

45 Helm Charts @gamussa | #devnexus | @ConfluentINc

46 https://cnfl.io/helm_video @gamussa | #devnexus | @ConfluentINc

47 Helm Charts is just a GO Templates. How Charts help with rolling update? @gamussa | #devnexus | @ConfluentINc

48 We will use StatefulSets with OrderedReady @gamussa | #devnexus | @ConfluentINc

49 We need SRE / Operator knowledge to manage the platform. You need Operator! @gamussa | #devnexus | @ConfluentINc

50 @gamussa | #devnexus | @ConfluentINc

Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under-replicated partitions 5. Upgrade the next broker

52 Will it Scale Spin up new brokers, connect workers easily Manual Rebalance required in v1.0 Determine balancing plan Execute balancing plan Monitor Resources @gamussa | #devnexus | @ConfluentINc

53 One More Thing…

54 Usual suspects Certified container images @gamussa | #devnexus | @ConfluentINc

55 @gamussa | #devnexus | @ConfluentINc

Thanks! @gamussa viktor@confluent.io https://slackpass.io/confluentcommunity #kubernetes @gamussa | @ #devnexus | @ConfluentINc

57