A presentation at SpringOne Platform by Pivotal in in Austin, TX, USA by Viktor Gamov
#SpringOne @s1p 1 Apache Kafka Event-Streaming Platform for .NET Developers October, 2019 @gamussa | #SpringOne | @ConfluentINc
2 @gamussa | #SpringOne | @ConfluentINc
3 I build highly scalable Hello World apps @gamussa | #springone | @ConfluentINc
4 A company is build on DATA FLOWS but All we have is DATA STORES @gamussa | #SpringOne | @ConfluentINc
5 Pre-Streaming @gamussa | #SpringOne | @ConfluentINc
6 @gamussa | #SpringOne | @ConfluentINc
8 New World Streaming first • DB/DWH + Many more distributed data systems • Monolith -> Microservices • Batch -> Real-time @gamussa | #SpringOne | @ConfluentINc
9 Origins in Stream Processing Java Apps with Kafka Streams or KSQL Serving Layer (Microservices, Elastic, etc.) High Throughput Continuous Streaming platform Computation @gamussa | #SpringOne | API based clustering @ConfluentINc
10 Streaming Platform Storage Pub / Sub Processing @gamussa | #SpringOne | @ConfluentINc
11 Storage @gamussa | #SpringOne | @ConfluentINc
12 Core Abstraction ● DB - table ● Hadoop - file ● Kafka - ? @gamussa | #SpringOne | @ConfluentINc
14 LOG @gamussa | #SpringOne | @ConfluentINc
15 The log is a simple idea New Old Messages are added at the end of the log @gamussa | #SpringOne | @ConfluentINc
16 The log is a simple idea New Old Messages are added at the end of the log @gamussa | #SpringOne | @ConfluentINc
17 Pub / Sub @gamussa | #SpringOne | @ConfluentINc
18 Time @gamussa | #SpringOne | @ConfluentINc
19 Time C1 @gamussa | C2 #SpringOne C3 | @ConfluentINc
20 Time A B hash(key) % numPartitions = N C D @gamussa | #SpringOne | @ConfluentINc
21 Time Messages will be produced in a round robin fashion @gamussa | #SpringOne | @ConfluentINc
22 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here Scan Viktor is here @gamussa | Scan #SpringOne | @ConfluentINc
23 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here @gamussa Viktor is here Scan | #SpringOne | Scan @ConfluentINc
24 Consumers have a position all of their own Ricardo is here Scan New Old Robin is here @gamussa | Viktor is here Scan #SpringOne | @ConfluentINc Scan
25 Only Sequential Access Old Read to offset & scan @gamussa | #SpringOne | @ConfluentINc New
CONSUMERS CONSUMER GROUP COORDINATOR CONSUMER GROUP
27 C @gamussa | #SpringOne | @ConfluentINc
28 CC C1 CC C2 @gamussa | #SpringOne | @ConfluentINc
29 @gamussa | #SpringOne | C C C C @ConfluentINc
30 @gamussa | #SpringOne | 0 1 2 3 @ConfluentINc
31 @gamussa | #SpringOne | 0 1 2 3 @ConfluentINc
32 @gamussa | #SpringOne | 0, 3 1 2 3 @ConfluentINc
33 Linearly Scalable Architecture Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!! Consumers @gamussa | #SpringOne | @ConfluentINc
34 Replicate to get fault leader msg Machine B Machine A @gamussa replicate | #SpringOne | msg @ConfluentINc
35 Partition Leadership and Replication Topic1 partition1 Topic1 partition1 Topic1 partition1 Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4 Broker 1 Broker 2 Topic1 partition4 Broker 3 Broker 4 Leader @gamussa | #SpringOne | @ConfluentINc Follower
36 Replication provides resiliency A replica takes over on machine failure @gamussa | #SpringOne | @ConfluentINc
37 Partition Leadership and Replication - node failure Topic1 partition1 Topic1 partition1 Topic1 partition1 Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4 Broker 1 Broker 2 Topic1 partition4 Broker 3 Broker 4 Leader @gamussa | #SpringOne | @ConfluentINc Follower
38 The log is a type of durable messaging system Similar to a traditional messaging system (ActiveMQ, Rabbit etc) but with: (a) Far better scalability (b) Built in fault tolerance / HA (c) Storage
Stop! Demo time! @gamussa | #SpringOne | @ConfluentINc
40 Processing @gamussa | #SpringOne | @ConfluentINc
41 Streaming is the toolset for dealing with events as they move! @gamussa | #SpringOne | @ConfluentINc
42 What exactly is Stream Processing? authorization_attempts @gamussa possible_fraud | #SpringOne | @ConfluentINc
43 What exactly is Stream Processing? possible_fraud authorization_attempts CREATE STREAM possible_fraud AS SELECT card_number, count() FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3; @gamussa | #SpringOne | @ConfluentINc
44 What exactly is Stream Processing? possible_fraud authorization_attempts CREATE STREAM possible_fraud AS SELECT card_number, count() FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3; @gamussa | #SpringOne | @ConfluentINc
45 What exactly is Stream Processing? possible_fraud authorization_attempts CREATE STREAM possible_fraud AS SELECT card_number, count() FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3; @gamussa | #SpringOne | @ConfluentINc
46 What exactly is Stream Processing? possible_fraud authorization_attempts CREATE STREAM possible_fraud AS SELECT card_number, count() FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3; @gamussa | #SpringOne | @ConfluentINc
47 What exactly is Stream Processing? possible_fraud authorization_attempts CREATE STREAM possible_fraud AS SELECT card_number, count() FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3; @gamussa | #SpringOne | @ConfluentINc
48 What exactly is Stream Processing? possible_fraud authorization_attempts CREATE STREAM possible_fraud AS SELECT card_number, count() FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count() > 3; @gamussa | #SpringOne | @ConfluentINc
49 Coding Sophistication Lower the bar to enter the world of streaming Core developers who use Java/Scala streams Core developers who don’t use Java/Scala Data engineers, architects, DevOps/SRE BI analysts User Population @gamussa | #SpringOne | @ConfluentINc
50 KSQL #FTW ksql> 1 UI POST /query CLI 2 @gamussa | #SpringOne 3 | REST @ConfluentINc 4 Headless
51 Interaction with Kafka KSQL Application Kafka (processing) (processing) Jva/KStreams, .NET (data) Does not run on Kafka brokers @gamussa Does not run on Kafka brokers | #SpringOne | @ConfluentINc
52 Find your local Meetup Group https://cnfl.io/kafka-meetups Grab Stream Processing books https://cnfl.io/book-bundle Join us in Slack http://cnfl.io/slack @gamussa | #SpringOne | @ConfluentINc
Thanks! @gamussa viktor@confluent.io @gamussa | @ #SpringOne | @ConfluentINc
When it comes time to choose a distributed messaging system, everyone knows the answer: Apache Kafka. But how about when you’re on the hook to choose a world-class, horizontally scalable stream data processing system? When you need not just publish and subscribe messaging, but also long-term storage, a flexible integration framework, and a means of deploying real-time stream processing applications at scale without having to integrate many different pieces of infrastructure yourself? The answer is still Apache Kafka.
In this talk, Viktor will give a rapid-fire review of the breadth of Kafka as a streaming data platform. You’ll see its internal architecture, including how it partitions messaging workloads in a fault-tolerant way, and how it provides message durability. Viktor will explain Kafka’s approach to pub/sub messaging and how the Confluent .NET client offers a framework for computation over streaming data.
Here’s what was said about this presentation on social media.
