Curious Case of Streaming Platform on Kubernetes @gamussa #devoops @confluentinc
A presentation at Devoops in October 2018 in St Petersburg, Russia by Viktor Gamov
Curious Case of Streaming Platform on Kubernetes @gamussa #devoops @confluentinc
#devkafkaops @gamussa #devoops @ @confluentinc
https://twitter.com/kelseyhightower/status/963413508300812295 @gamussa #devoops @ @confluentinc
https://twitter.com/kelseyhightower/status/963414038603427840 @gamussa #devoops @ @confluentinc
@gamussa #devoops @ @confluentinc
Why streams? Tao of the streaming platform
@gamussa #devoops @ @confluentinc
@
@gamussa #devoops @ @confluentinc
@
Streaming Platform Architecture Application Application Application Native Client library Kafka Streams Load Balancer * REST Proxy Schema Registry Kafka Brokers @gamussa Kafka Connect Zookeeper Nodes #devoops @ @confluentinc
Bare minimum Kafka Brokers @gamussa Zookeeper Nodes #devoops @ @confluentinc
Shard data to get scalability Producer (1) Producer (2) Producer (3) Messages are sent to different partitions Cluster of machines Partitions live on different machines @gamussa #devoops @confluentinc
Linearly Scalable Architecture Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!! Consumers @gamussa #devoops @confluentinc
Replicate to get fault tolerance leader msg Machine A @gamussa Machine B replicate #devoops msg @confluentinc
Replication provides resiliency A ‘replica’ takes over on machine failure @gamussa #devoops @confluentinc
Microservices Docker Kubernetes Monolith @gamussa #devoops @ @confluentinc
https://twitter.com/sahrizv/status/1018184792611827712 @gamussa #devoops @ @confluentinc
Kafka as Microservices Backbone HTTP etc Online Offline Billing Inventory @gamussa Fulfillment Finance #devoops @confluentinc @ Fraud
Stateful Workloads Key Concepts
Refresher - Kubernetes Architecture kubectl https://thenewstack.io/kubernetes-an-overview/ @gamussa #devoops @ @confluentinc
Controller Brain behind Kubernetes resources e.g. replication controller, namespace controller etc. @gamussa #devoops @ @confluentinc
Pod Basic Unit of Deployment in Kubernetes A collection of containers sharing: Namespace Network Volumes @gamussa #devoops @ @confluentinc
Custom Resource Definition(CRD) Extend existing Kubernetes API API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa #devoops @ StatefulSet @confluentinc
Custom Resource Definition(CRD) Usually works together Custom Controller (operator pattern) API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance @gamussa #devoops @ StatefulSet @confluentinc
Custom Resource Definition(CRD) Users can create and access Customer Resources with kubectl, just as they do for built-in API StatefulSet ReplicaSet ... CRD Controller StatefulSet Controller ReplicaSet Controller ... Custom Controller ReplicaSet ... Custom Resource Instance StatefulSet resources like pods. @gamussa #devoops @ @confluentinc
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) Both PV and PVC are ‘resources’ @gamussa #devoops @ @confluentinc
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV @gamussa #devoops @ @confluentinc
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVC is a request for storage by a User @gamussa #devoops @ @confluentinc
Storage Persistent Volume (PV) & Persistent Volume Claim (PVC) PVCs consume PV @gamussa #devoops @ @confluentinc
StatefulSet Rely on Headless Service to provide network identity Headless Service Ideal for highly available stateful workloads @gamussa Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc
StatefulSet Rely on Headless Service to provide network identity @gamussa Headless Service Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc
StatefulSet Ideal for highly available stateful workloads @gamussa Headless Service Pod-0 #devoops @ Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes @confluentinc
Workloads Deployment Enter Helm #ihateyaml
Helm Charts @gamussa #devoops @ @confluentinc
Helm Charts @gamussa #devoops @ @confluentinc
Helm Charts @gamussa #devoops @ @confluentinc
Kubernetes Operator Custom Resource Definition (CRDs) Extending K8s Controller @gamussa #devoops @ @confluentinc
Kubernetes Operator Embedded with operational knowledge of both data software and Kubernetes Backup/restore Scale up/down Rebalance data Regular health checks @gamussa #devoops @ @confluentinc
Bare minimum Kafka Brokers @gamussa Zookeeper Nodes #devoops @ @confluentinc
ZK & Kafka install PVC for Storage Uses ZK Headless Svc StatefulSet for 3-node zk PVC for Storage Optional Pod Anti-Affinity to spread the ZK ensemble across nodes. StatefulSet for n-node Kafka Headless Service A group of NodePort Services for external traffic ConfigMap for Prometheus JMX exporter ConfigMap for Prometheus JMX exporter @gamussa #devoops @ @confluentinc
Kafka Kubernetes journey: Step 1 https://cnfl.io/cp-helm @gamussa #devoops @ @confluentinc
Let’s see some YAML! @gamussa #devoops @confluentinc
Step 2: Operator Deploy and Manage your production streaming platform with Confluent Operator. Automated Provisioning Platform Operations Resiliency Monitoring @gamussa #devoops @ @confluentinc
Confluent Platform Reference Architecture Each Confluent Platform component has specific characteristics: Security (SSL certificates) DNS names and zones Application Application Application Native Client library Kafka Streams Load Balancer * Schema Registry REST Proxy Kafka Connect Host selection Fault tolerance Kafka Brokers Scaling @gamussa #devoops @ @confluentinc Zookeeper Nodes
Confluent Operator: Automated Provisioning Load Balancer Kafka Pod Kafka Pod Kafka Pod Storage @gamussa #devoops @ @confluentinc
Confluent Operator: Scale Horizontally Automate scaling: Spin up new broker pod(s) Distribute partitions to the new broker(s) Determine balancing plan Execute balancing plan Monitor resources @gamussa #devoops @ @confluentinc
Confluent Operator: Rolling Upgrade Automated rolling upgrade with no downtime for Kafka. Stop broker Wait for leader election to complete Start broker with new version Wait for zero under-replicatedpartitions Repeat @gamussa #devoops @ @confluentinc
Will it fly? Let’s see @gamussa #devoops @confluentinc
Confluent Operator Automate provisioning Scale your Kafkas and CP clusters elastically Monitor SLAs through Confluent Control Center or Prometheus Operate at scale with enterprise support from Confluent @gamussa #devoops @ @confluentinc
Future
Vision of the future vs. @gamussa #devoops @confluentinc
Resources and Next Steps https://cnfl.io/cp-helm https://cnfl.io/k8s https://slackpass.io/confluentcommunity #kubernetes @gamussa #devoops @confluentinc
Thanks! @gamussa viktor@confluent.io We are hiring! https://www.confluent.io/careers/ @gamussa #devoops @ @confluentinc
