Kafka on Kubernetes: Just because you can, doesn’t mean you should!

Kafka on Kubernetes: Just because you can, doesn’t mean you should! July, 2019 / New York City, 2019 @gamussa @gamussa || #DockerNYC #DockerNYC || @ConfluentINc @ConfluentINc

2 @gamussa | #DockerNYC | @ConfluentINc

3 I build highly scalable Hello World apps @gamussa | #DockerNYC | @ConfluentINc

4 A company is built on DATA FLOWS but All we have is DATA STORES @gamussa | #DockerNYC | @ConfluentINc

5 Pre-streaming @gamussa | #DockerNYC | @ConfluentINc

6 @gamussa | #DockerNYC | @ConfluentINc

7 New World Streaming first • DB/DWH + Many more distributed data systems • Monolith -> Microservices • Batch -> Real-time @gamussa | #DockerNYC | @ConfluentINc

8 Evolution of #devkafkaops Ansible Docker Kubernetes Shell scripts @gamussa | #DockerNYC | @ConfluentINc

9 https://twitter.com/claesmogren/status/1108456688175079424 https://twitter.com/QuinnyPig/status/1150927901782499330 @gamussa | #DockerNYC | @ConfluentINc

10 🙋 @gamussa | #DockerNYC | @ConfluentINc

11 Who run stateless apps in Kubernetes? 🙋 Who thinks it’s a good idea? Who run stateful app in Kubernetes? Who thinks it’s a good idea? @gamussa | #DockerNYC | @ConfluentINc

12 kafkaesque world of Kafka on Kubernetes @gamussa | #DockerNYC | @ConfluentINc

13 #devkafkaops Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades @gamussa | #DockerNYC | @ConfluentINc

14 But I just want to deploy kafka @gamussa | #DockerNYC | @ConfluentINc

15 Kubernetes Fundamentals @gamussa | #DockerNYC | @ConfluentINc

16 https://twitter.com/sahrizv/status/1018184792611827712 @gamussa | #DockerNYC | @ConfluentINc

17 @gamussa | #DockerNYC | @ConfluentINc

18 Kubernetes ●Schedules and allocates resources ●Networking between Pods ●Storage ●Service Discovery @gamussa | #DockerNYC | @ConfluentINc

19 https://twitter.com/claesmogren/status/1108456688175079424 @gamussa | #DockerNYC | @ConfluentINc

20 Kubernetes Architecture kubectl https://thenewstack.io/kubernetes-an-overview/ @gamussa | #DockerNYC | @ConfluentINc

21 Pod • Basic Unit of Deployment in Kubernetes • A collection of containers sharing: • Namespace • Network • Volumes @gamussa | #DockerNYC | @ConfluentINc

22 Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • Both PV and PVC are ‘resources’ @gamussa | #DockerNYC | @ConfluentINc

23 Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV @gamussa | #DockerNYC | @ConfluentINc

24 Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PVC is a request for storage by a User @gamussa | #DockerNYC | @ConfluentINc

25 Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PVCs consume PV @gamussa | #DockerNYC | @ConfluentINc

26 Stateful Workloads @gamussa | #DockerNYC | @ConfluentINc

27 We just need to deploy Kafka on Kubernetes @gamussa | #DockerNYC | @ConfluentINc

28 We will use confluentinc/cp-helmcharts @gamussa | #DockerNYC | @ConfluentINc

29 Workloads Deployment @gamussa | #DockerNYC | @ConfluentINc

30 Helm Charts @gamussa | #DockerNYC | @ConfluentINc

31 Helm Charts @gamussa | #DockerNYC | @ConfluentINc

32 Helm Charts @gamussa | #DockerNYC | @ConfluentINc

33 https://cnfl.io/helm_video @gamussa | #DockerNYC | @ConfluentINc

34 Helm Charts is just a GO Templates. How Charts help with rolling update? @gamussa | #DockerNYC | @ConfluentINc

35 Be like Justin! @gamussa | #DockerNYC | @ConfluentINc

Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero underreplicated partitions 5. Upgrade the next broker @gamussa | #DockerNYC | @ConfluentINc 36

37 We will use StatefulSets with OrderedReady @gamussa | #DockerNYC | @ConfluentINc

38 We need SRE / Operator knowledge to manage the platform. You need Operator! @gamussa | #DockerNYC | @ConfluentINc

39 @gamussa | #DockerNYC | @ConfluentINc

40 Show me your Operator @gamussa | #DockerNYC | @ConfluentINc

41 KLyfft - Kafka app on Kubernetes 2 Web Apps powered by a microservice each Apps are a ride sharing driver and rider location tracking Rider hails a ride and the driver drives towards the rider Real time location events powered by Kafka on Kubernetes Rider @gamussa | #DockerNYC | @ConfluentINc Driver

DO KAFKA ON KUBERNETES DEMO AND EVERYONE LOOSES THEIR MINDS @gamussa | #DockerNYC | @ConfluentINc 42

43 @gamussa | #DockerNYC | @ConfluentINc

44 Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components @gamussa | #DockerNYC | @ConfluentINc

45 Want to see advanced deployment? vs. @gamussa | #DockerNYC | @ConfluentINc

46 http://gamov.dev/ksummit_nyc19_k8s @gamussa | #DockerNYC | @ConfluentINc

Gamov30 30% OFF*, duh! *Standard Priced Conference pass

Thanks! @gamussa viktor@confluent.io https://slackpass.io/confluentcommunity #kubernetes @gamussa | @ #DockerNYC | @ConfluentINc

49