Kafka on Kubernetes: Just because you can, doesn’t mean you should! July, 2019 / New York City, 2019 @gamussa @gamussa
|| #DockerNYC #DockerNYC
||
@ConfluentINc @ConfluentINc
4
A company is built on DATA FLOWS but All we have is DATA STORES @gamussa
|
#DockerNYC
|
@ConfluentINc
7
New World Streaming first • DB/DWH + Many more distributed data systems • Monolith -> Microservices • Batch -> Real-time
@gamussa
|
#DockerNYC
|
@ConfluentINc
8
Evolution of #devkafkaops
Ansible
Docker
Kubernetes
Shell scripts
@gamussa
|
#DockerNYC
|
@ConfluentINc
9
https://twitter.com/claesmogren/status/1108456688175079424 https://twitter.com/QuinnyPig/status/1150927901782499330
@gamussa
|
#DockerNYC
|
@ConfluentINc
11
Who run stateless apps in Kubernetes?
🙋
Who thinks it’s a good idea? Who run stateful app in Kubernetes? Who thinks it’s a good idea? @gamussa
|
#DockerNYC
|
@ConfluentINc
13
#devkafkaops
Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades
@gamussa
|
#DockerNYC
|
@ConfluentINc
18
Kubernetes ●Schedules and allocates resources ●Networking between Pods ●Storage ●Service Discovery
@gamussa
|
#DockerNYC
|
@ConfluentINc
20
Kubernetes Architecture kubectl
https://thenewstack.io/kubernetes-an-overview/ @gamussa
|
#DockerNYC
|
@ConfluentINc
21
Pod • Basic Unit of Deployment in Kubernetes • A collection of containers sharing: • Namespace • Network • Volumes @gamussa
|
#DockerNYC
|
@ConfluentINc
22
Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • Both PV and PVC are ‘resources’
@gamussa
|
#DockerNYC
|
@ConfluentINc
23
Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PV is a piece of storage that is provisioned dynamic or static of any individual pod that uses the PV
@gamussa
|
#DockerNYC
|
@ConfluentINc
24
Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PVC is a request for storage by a User
@gamussa
|
#DockerNYC
|
@ConfluentINc
25
Storage • Persistent Volume (PV) & Persistent Volume Claim (PVC) • PVCs consume PV
@gamussa
|
#DockerNYC
|
@ConfluentINc
34
Helm Charts is just a GO Templates. How Charts help with rolling update?
@gamussa
|
#DockerNYC
|
@ConfluentINc
Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero underreplicated partitions 5. Upgrade the next broker @gamussa
|
#DockerNYC
|
@ConfluentINc
36
38
We need SRE / Operator knowledge to manage the platform. You need Operator!
@gamussa
|
#DockerNYC
|
@ConfluentINc
41
KLyfft - Kafka app on Kubernetes 2 Web Apps powered by a microservice each Apps are a ride sharing driver and rider location tracking Rider hails a ride and the driver drives towards the rider Real time location events powered by Kafka on Kubernetes Rider
@gamussa
|
#DockerNYC
|
@ConfluentINc
Driver
DO KAFKA ON KUBERNETES DEMO
AND EVERYONE LOOSES THEIR MINDS @gamussa
|
#DockerNYC
|
@ConfluentINc
42
44
Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components
@gamussa
|
#DockerNYC
|
@ConfluentINc
Gamov30 30% OFF*, duh!
*Standard Priced Conference pass
Thanks! @gamussa viktor@confluent.io
https://slackpass.io/confluentcommunity #kubernetes @gamussa
|
@ #DockerNYC
|
@ConfluentINc
Kafka on Kubernetes: Just because you can, doesn’t mean you should! July, 2019 / New York City, 2019 @gamussa @gamussa
|| #DockerNYC #DockerNYC
||
@ConfluentINc @ConfluentINc