Streams Must Flow, Deploying fault-tolerant stream processing applications to Kubernetes

Streams Must Flow: fault-tolerant stream processing apps on Kubernetes April, 2019 / New York, NY @gamussa | #cloudnativenyc | @ConfluentINc

2 Special thanks! @gwenshap @gamussa @MatthiasJSax | #cloudnativenyc | @ConfluentINc

3 Agenda Kafka Streams 101 How do Kafka Streams applications scale? Stateful Workloads Recommendations for Kafka Streams @gamussa | #cloudnativenyc | @ConfluentINc

4 Kafka Streams – 101 Your App @gamussa | #cloudnativenyc | @ConfluentINc Other Systems Kafka Connect Kafka Connect Other Systems Kafka Streams

5 Stock Trade Stats Example KStream<String, Trade> source = builder.stream(STOCK_TOPIC); KStream<Windowed<String>, TradeStats> stats = source .groupByKey() .windowedBy(TimeWindows.of(5000).advanceBy(1000)) .aggregate(TradeStats::new, (k, v, tradestats) -> tradestats.add(v), Materialized.<~>as(“trade-aggregates”) .withValueSerde(new TradeStatsSerde())) .toStream() .mapValues(TradeStats::computeAvgPrice); stats.to(STATS_OUT_TOPIC, Produced.keySerde(WindowedSerdes.timeWindowedSerdeFrom(String.class))); @gamussa | #cloudnativenyc | @ConfluentINc

6 Stock Trade Stats Example KStream<String, Trade> source = builder.stream(STOCK_TOPIC); KStream<Windowed<String>, TradeStats> stats = source .groupByKey() .windowedBy(TimeWindows.of(5000).advanceBy(1000)) .aggregate(TradeStats::new, (k, v, tradestats) -> tradestats.add(v), Materialized.<~>as(“trade-aggregates”) .withValueSerde(new TradeStatsSerde())) .toStream() .mapValues(TradeStats::computeAvgPrice); stats.to(STATS_OUT_TOPIC, Produced.keySerde(WindowedSerdes.timeWindowedSerdeFrom(String.class))); @gamussa | #cloudnativenyc | @ConfluentINc

7 Stock Trade Stats Example KStream<String, Trade> source = builder.stream(STOCK_TOPIC); KStream<Windowed<String>, TradeStats> stats = source .groupByKey() .windowedBy(TimeWindows.of(5000).advanceBy(1000)) .aggregate(TradeStats::new, (k, v, tradestats) -> tradestats.add(v), Materialized.<~>as(“trade-aggregates”) .withValueSerde(new TradeStatsSerde())) .toStream() .mapValues(TradeStats::computeAvgPrice); stats.to(STATS_OUT_TOPIC, Produced.keySerde(WindowedSerdes.timeWindowedSerdeFrom(String.class))); @gamussa | #cloudnativenyc | @ConfluentINc

8 Stock Trade Stats Example KStream<String, Trade> source = builder.stream(STOCK_TOPIC); KStream<Windowed<String>, TradeStats> stats = source .groupByKey() .windowedBy(TimeWindows.of(5000).advanceBy(1000)) .aggregate(TradeStats::new, (k, v, tradestats) -> tradestats.add(v), Materialized.<~>as(“trade-aggregates”) .withValueSerde(new TradeStatsSerde())) .toStream() .mapValues(TradeStats::computeAvgPrice); stats.to(STATS_OUT_TOPIC, Produced.keySerde(WindowedSerdes.timeWindowedSerdeFrom(String.class))); @gamussa | #cloudnativenyc | @ConfluentINc

9 Topologies builder.stream Source Node state stores source.groupByKey .windowedBy(…) .aggregate(…) Processor Node mapValues() Processor Node streams Sink Node to(…) @gamussa | #cloudnativenyc | @ConfluentINc Processor Topology

How Do Kafka Streams Application Scale? @gamussa | #cloudnativenyc | @ConfluentINc

11 Partitions, Tasks, and Consumer Groups input topic Task executes processor topology One consumer group: can be executed with 1 - 4 threads on 1 - 4 machines 4 input topic partitions => 4 tasks result topic @gamussa | #cloudnativenyc | @ConfluentINc

12 Scaling with State “no state” Trade Stats App Instance 1 @gamussa | #cloudnativenyc | @ConfluentINc

Scaling with State “no state” Trade Stats App Trade Stats App Instance 1 @gamussa Instance 2 | #cloudnativenyc | @ConfluentINc 13

14 Scaling with State “no state” Trade Stats App Trade Stats App Instance 1 @gamussa Instance 2 | #cloudnativenyc | Trade Stats App Instance 3 @ConfluentINc

15 Scaling and FaultTolerance Two Sides of Same Coin @gamussa | #cloudnativenyc | @ConfluentINc

16 Fault-Tolerance Trade Stats App Trade Stats App Instance 1 @gamussa | Instance 2 #cloudnativenyc | Trade Stats App Instance 3 @ConfluentINc

17 Fault-Tolerant State State Updates Input Topic Changelog Topic Result Topic @gamussa | #cloudnativenyc | @ConfluentINc

18 Migrate State Trade Stats App Instance 1 Trade Stats App Instance 2 restore Changelog Topic @gamussa | #cloudnativenyc | @ConfluentINc

19 Recovery Time • Changelog topics are log compacted • Size of changelog topic linear in size of state • Large state implies high recovery times @gamussa | #cloudnativenyc | @ConfluentINc

20 Stateful Workloads @gamussa | #cloudnativenyc | @ConfluentINc

21 StatefulSet ● Rely on Headless Headless Service Service to provide network identity Pod-0 ● Ideal for highly Pod-1 Pod-2 Containers Containers Containers Volumes Volumes Volumes available stateful workloads @gamussa | #cloudnativenyc | @ConfluentINc

Recommendations for Kafka Streams @gamussa | #cloudnativenyc | @ConfluentINc

23 Stock Stats App Stock Stats App Stock Stats App Kafka Streams Kafka Streams Kafka Streams Instance 1 Instance 2 Instance 3 @gamussa | #cloudnativenyc | @ConfluentINc

24 WordCount App WordCount App WordCount App Kafka Streams Kafka Streams Kafka Streams Instance 1 Instance 2 Instance 3 @gamussa | #cloudnativenyc | @ConfluentINc

25 StatefulSets are new and complicated. We don’t need them. @gamussa | #cloudnativenyc | @ConfluentINc

26 Recovering state takes time. Statelful is faster. @gamussa | #cloudnativenyc | @ConfluentINc

27 But I’ll want to scale-out and back anyway. @gamussa | #cloudnativenyc | @ConfluentINc

28 @gamussa | #cloudnativenyc | @ConfluentINc

29 I don’t really trust my storage admin anyway @gamussa | #cloudnativenyc | @ConfluentINc

30 Recommendations ● Keep changelog shards small ● If you trust your storage: Use StatefulSets ● Use anti-affinity when possible ● Use “parallel” pod management

31 🛑 Stop! Demo time! @gamussa | #cloudnativenyc | @ConfluentINc

32 Summary Kafka Streams has recoverable state, that gives streams apps easy elasticity and high availability Kubernetes makes it easy to scale applications It also has StatefulSets for applications with state @gamussa | #cloudnativenyc | @ConfluentINc

33 Summary Now you know how to deploy Kafka Streams on Kubernetes and take advantage on all the scalability and highavailability capabilities @gamussa | #cloudnativenyc | @ConfluentINc

34 But what about Kafka itself? @gamussa | #cloudnativenyc | @ConfluentINc

35 Confluent Operator Automate provisioning Scale your Kafkas and CP clusters elastically Monitoring with Confluent Control Center or Prometheus Operate at scale with enterprise support from Confluent @gamussa | #cloudnativenyc | @ConfluentINc

36 Resources and Next Steps https://cnfl.io/helm_video https://cnfl.io/cp-helm https://cnfl.io/k8s https://slackpass.io/confluentcommunity #kubernetes @gamussa | #cloudnativenyc | @ConfluentINc

Thanks! @gamussa viktor@confluent.io We are hiring! https://www.confluent.io/careers/ @gamussa | @ #cloudnativenyc | @ConfluentINc

38