Kafka на Kubernetes: загадочная история создания облачной платформы поточной обработки данных

A presentation at Devoops in October 2018 in St Petersburg, Russia by Viktor Gamov

Slide 1

Slide 1

Curious Case of Streaming Platform on Kubernetes @gamussa #devoops @confluentinc

Slide 2

Slide 2

Slide 3

Slide 3

Slide 4

Slide 4

#devkafkaops @gamussa #devoops @ @confluentinc

Slide 5

Slide 5

https://twitter.com/kelseyhightower/status/963413508300812295 @gamussa #devoops @ @confluentinc

Slide 6

Slide 6

https://twitter.com/kelseyhightower/status/963414038603427840 @gamussa #devoops @ @confluentinc

Slide 7

Slide 7

@gamussa #devoops @ @confluentinc

Slide 8

Slide 8

Why streams? Tao of the streaming platform

Slide 9

Slide 9

@gamussa #devoops @ @confluentinc

Slide 10

Slide 10

@

Slide 11

Slide 11

@gamussa #devoops @ @confluentinc

Slide 12

Slide 12

@

Slide 13

Slide 13

Streaming Platform Architecture Application Application Application Native Client library Kafka Streams Load Balancer * REST Proxy Schema Registry Kafka Brokers @gamussa Kafka Connect Zookeeper Nodes #devoops @ @confluentinc

Slide 14

Slide 14

Bare minimum Kafka Brokers @gamussa Zookeeper Nodes #devoops @ @confluentinc

Slide 15

Slide 15

Shard data to get scalability Producer (1) Producer (2) Producer (3) Messages are sent to different partitions Cluster of machines Partitions live on different machines @gamussa #devoops @confluentinc

Slide 16

Slide 16

Linearly Scalable Architecture Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!! Consumers @gamussa #devoops @confluentinc

Slide 17

Slide 17

Replicate to get fault tolerance leader msg Machine A @gamussa Machine B replicate #devoops msg @confluentinc

Slide 18

Slide 18

Replication provides resiliency A ‘replica’ takes over on machine failure @gamussa #devoops @confluentinc

Slide 19

Slide 19

Microservices Docker Kubernetes Monolith @gamussa #devoops @ @confluentinc

Slide 20

Slide 20

https://twitter.com/sahrizv/status/1018184792611827712 @gamussa #devoops @ @confluentinc

Slide 21

Slide 21

Kafka as Microservices Backbone HTTP etc Online Offline Billing Inventory @gamussa Fulfillment Finance #devoops @confluentinc @ Fraud

Slide 22

Slide 22

Stateful Workloads Key Concepts

Slide 23

Slide 23

Refresher - Kubernetes Architecture kubectl https://thenewstack.io/kubernetes-an-overview/ @gamussa #devoops @ @confluentinc

Slide 24

Slide 24

Controller Brain behind Kubernetes resources e.g. replication controller, namespace controller etc. @gamussa #devoops @ @confluentinc

Slide 25

Slide 25

Pod Basic Unit of Deployment in Kubernetes A collection of containers sharing: Namespace Network Volumes @gamussa #devoops @ @confluentinc

Slide 26

Slide 26

Custom Resource Definition(CRD) Extend existing Kubernetes API API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa #devoops @ StatefulSet @confluentinc

Slide 27

Slide 27

Custom Resource Definition(CRD) Usually works together Custom Controller (operator pattern) API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa #devoops @ StatefulSet @confluentinc

Slide 28

Slide 28

Custom Resource Definition(CRD) Users can create and access Customer Resources with kubectl, just as they do for built-in API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance StatefulSet resources like pods. @gamussa #devoops @ @confluentinc

Slide 29

Slide 29

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) Both PV and PVC are ‘resources’ @gamussa #devoops @ @confluentinc

Slide 30

Slide 30

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV @gamussa #devoops @ @confluentinc

Slide 31

Slide 31

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVC is a request for storage by a User @gamussa #devoops @ @confluentinc

Slide 32

Slide 32

Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVCs consume PV @gamussa #devoops @ @confluentinc

Slide 33

Slide 33

StatefulSet Rely on Headless Service to provide network identity Headless Service Ideal for highly available stateful workloads @gamussa Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc

Slide 34

Slide 34

StatefulSet Rely on Headless Service to provide network identity @gamussa Headless Service Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc

Slide 35

Slide 35

StatefulSet Ideal for highly available stateful workloads @gamussa Headless Service Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc

Slide 36

Slide 36

Workloads Deployment Enter Helm #ihateyaml

Slide 37

Slide 37

Helm Charts @gamussa #devoops @ @confluentinc

Slide 38

Slide 38

Helm Charts @gamussa #devoops @ @confluentinc

Slide 39

Slide 39

Helm Charts @gamussa #devoops @ @confluentinc

Slide 40

Slide 40

Kubernetes Operator Custom Resource Definition (CRDs) Extending K8s Controller @gamussa #devoops @ @confluentinc

Slide 41

Slide 41

Kubernetes Operator Embedded with operational knowledge of both data software and Kubernetes Backup/restore Scale up/down Rebalance data Regular health checks @gamussa #devoops @ @confluentinc

Slide 42

Slide 42

Bare minimum Kafka Brokers @gamussa Zookeeper Nodes #devoops @ @confluentinc

Slide 43

Slide 43

ZK & Kafka install PVC for Storage Uses ZK Headless Svc StatefulSet for 3-node zk PVC for Storage Optional Pod Anti-Affinity to spread the ZK ensemble across nodes. StatefulSet for n-node Kafka Headless Service A group of NodePort Services for external traffic ConfigMap for Prometheus JMX exporter ConfigMap for Prometheus JMX exporter @gamussa #devoops @ @confluentinc

Slide 44

Slide 44

Kafka Kubernetes journey: Step 1 https://cnfl.io/cp-helm @gamussa #devoops @ @confluentinc

Slide 45

Slide 45

Let’s see some YAML! @gamussa #devoops @confluentinc

Slide 46

Slide 46

Step 2: Operator Deploy and Manage your production streaming platform with Confluent Operator. Automated Provisioning Platform Operations Resiliency Monitoring @gamussa #devoops @ @confluentinc

Slide 47

Slide 47

Confluent Platform Reference Architecture Each Confluent Platform component has specific characteristics: Security (SSL certificates) DNS names and zones Application Application Application Native Client library Kafka Streams Load Balancer * Schema Registry REST Proxy Kafka Connect Host selection Fault tolerance Kafka Brokers Scaling @gamussa #devoops @ @confluentinc Zookeeper Nodes

Slide 48

Slide 48

Confluent Operator: Automated Provisioning Load Balancer Kafka Pod Kafka Pod Kafka Pod Storage @gamussa #devoops @ @confluentinc

Slide 49

Slide 49

Confluent Operator: Scale Horizontally Automate scaling: Spin up new broker pod(s) Distribute partitions to the new broker(s) Determine balancing plan Execute balancing plan Monitor resources @gamussa #devoops @ @confluentinc

Slide 50

Slide 50

Confluent Operator: Rolling Upgrade Automated rolling upgrade with no downtime for Kafka. Stop broker Wait for leader election to complete Start broker with new version Wait for zero under-replicatedpartitions Repeat @gamussa #devoops @ @confluentinc

Slide 51

Slide 51

Will it fly? Let’s see @gamussa #devoops @confluentinc

Slide 52

Slide 52

Confluent Operator Automate provisioning Scale your Kafkas and CP clusters elastically Monitor SLAs through Confluent Control Center or Prometheus Operate at scale with enterprise support from Confluent @gamussa #devoops @ @confluentinc

Slide 53

Slide 53

Future

Slide 54

Slide 54

Vision of the future vs. @gamussa #devoops @confluentinc

Slide 55

Slide 55

Resources and Next Steps https://cnfl.io/cp-helm https://cnfl.io/k8s https://slackpass.io/confluentcommunity #kubernetes @gamussa #devoops @confluentinc

Slide 56

Slide 56

Thanks! @gamussa viktor@confluent.io We are hiring! https://www.confluent.io/careers/ @gamussa #devoops @ @confluentinc