Event-Driven Architectures for Spring Developers

1 #SpringOne @s1p Event-Driven Architectures for Spring Developers Viktor Gamov Developer Advocate, Confluent @gamussa Gary Russell Engineer, Pivotal @gprussell

2 Why Event Streaming ? @gamussa | #SpringOne | @gprussell

3 The world is changing.

4 The New Business Reality Past Today Technology was a support function Technology is the business Innovation required for growth Innovation required for survival Running the business on yesterday’s data Yesterday’s data = failure. was “good enough” Modern, real-time data infrastructure is required. @gamussa | #SpringOne | @gprussell

5 Taxis become Software 2 min @gamussa | #SpringOne | @gprussell

6 Transportation Then Hardware product Up-front purchase Opaque No data Now Hardware, Software, and Global Internet Service The world is changing. On-demand Real-time visibility Built on a foundation of data

7 What enables this transformation? @gamussa | #SpringOne | @gprussell

8 Cloud Machine Learning Mobile Event Streaming Rethink Data Centers Rethink Decision Making Rethink User Experience Rethink Data @gamussa | #SpringOne | @gprussell

9 Do you see me? Or: Would you blindly cross the street with traffic information that is 5 minutes old?

10 Transportation Banking Retail ETA Fraud detection Real-time inventory Real-time sensor diagnostics Trading and risk systems Real-time POS reporting Driver-rider match Mobile applications / customer experience Personalization @gamussa | #SpringOne Entertainment Real-time recommendations Personalized news feed In-app purchases | @gprussell

11 This is a fundamental paradigm shift… Cloud Future of the datacenter Infrastructure as code Event Streaming Future of data Data as continuous stream of events @gamussa | #SpringOne | @gprussell

17 12 The Event Streaming Paradigm @gamussa | #SpringOne | @gprussell

13 Two Problems in Application Infrastructure What’s the state of the world? What’s happening in the world? Solution: Solution: Databases Messaging, RPC, ETL, etc. @gamussa | #SpringOne | @gprussell

14 ETL/Data Integration Batch Messaging High Throughput Expensive Time Consuming Fast (Low Latency) Difficult to Scale Durable No Persistence Persistent Data Loss Maintains Order No Replay @gamussa | #SpringOne | @gprussell

15 ETL/Data Integration Batch Messaging High Throughput Expensive Time Consuming Fast (Low Latency) Difficult to Scale Durable No Persistence Persistent Data Loss Maintains Order No Replay Transient Messages Stored records @gamussa | #SpringOne | @gprussell

16 @gamussa | #SpringOne | @gprussell

17 ETL/Data Integration Batch Event Streaming Paradigm High High Throughput Throughput Expensive Time Consuming Durable Durable Fast Fast (Low (Low Latency) Latency) Messaging Difficult to Scale No Persistence Replay Persistent Persistent Data Loss Maintains Maintains Order Order No Replay Stored records Transient Messages @gamussa | #SpringOne | @gprussell

18 Event Streaming Paradigm To rethink data as neither stored records nor transient messages, but instead as a continuously updating Stream of Events @gamussa | #SpringOne | @gprussell

19 24 An Event records the fact that something happened A good was sold An invoice was issued @gamussa | A payment was made #SpringOne | @gprussell A new customer registered

20 25 A Stream represents history as a sequence of Events @gamussa | #SpringOne | @gprussell

21 26 Events change the way we think Monolithic Approach Event-First Approach ● a database ● an event ● a stream ● a variable ● a singleton ● an RPC ● a ‘data’ flow ● a stream processor Orders Service Payments Service Orders Service Order Validation Service Customers Service request response event streams Email Notification Service DB @gamussa | #SpringOne | @gprussell Tax Service

22 27 An Event Streaming Platform gives you three key functionalities Publish & Subscribe to Events @gamussa Store Events | #SpringOne Process & Analyze Events | @gprussell

30 23 Event-Driven Apps, with Historical Context Real-Time Inventory Real-Time Fraud Detection Real-Time Customer 360 Machine Learning Models Real-Time Data Transformation Event Streaming Platform … ✓ Real-time but also persistent ✓ Elastic, scalable, reliable ✓ High throughput, low latency ✓ All apps and systems can now speak to each other for a complete view of data Universal Event Pipeline Data Stores Logs @gamussa 3rd Party Apps | #SpringOne Custom Apps/Microservices | @gprussell

24 Why Combine Real-time With Historical Context? Contextual Event-Driven App Event-Driven App (Location Tracking) (ETA) 2 min Only Real-time Events Messaging Queues and Event Streaming Platforms can do this Whereisismy mydriver? driver? Where @gamussa | #SpringOne Real-time combined with stored data Only Event Streaming Platforms can do this When When will will my my driver driver get get here? here? | @gprussell

33 25 How to Build Event Streaming Architectures With Kafka @gamussa | #SpringOne | @gprussell

26 is a distributed event streaming platform Publish & Subscribe to Events @gamussa Store Events | #SpringOne Process & Analyze Events | @gprussell

35 27 01 Stream your data in real-time as Events 02 03 Store your Event Streams @gamussa | #SpringOne Process & Analyze your Events Streams | @gprussell

36 28 From apps, microservices Use a Kafka producer client from your favorite language … and many more 01 https://gamov.dev/kafka-clients-demo Stream your data in real-time as Events From/to other systems Use Kafka Connect plus a Connector for your system … and many more confluent.io/hub @gamussa | #SpringOne @gamussa | | @gprussell #SpringOne | @gprussell

29 From apps, microservices: producer example network write @Service class Producer @Autowired constructor(private val kafkaTemplate: KafkaTemplate<String, User>) { @Value(“${topic.name}”) lateinit var TOPIC: String; internal fun sendMessage(user: User) { this.kafkaTemplate.send(this.TOPIC, user.getName(), user) println(String.format(“Produced user !-> %s”, user)) } } … and more

30 From/to other systems: Kafka Connect and more Tip: Great option to gradually move workloads to Kafka while keeping production running!

31 Kafka Connect ● Deployed standalone (development) or as a distributed cluster (production) ● Elastic service that works on bare-metal, VMs, containers, Kubernetes, … ● The individual ‘Connector’ determines delivery guarantees, e.g., exactly-once VM VM

32 Single Message Transforms for real-time ETL Ingress: modify an Event before storing Egress: modify an Event on its way out ● Obfuscate sensitive information, e.g. PII ● Route high-priority events to faster stores ● Add origin of event for lineage tracking ● Direct events to different Elasticsearch ● Remove unnecessary data fields ● … and more { user: ab123, gender: female, ip: 1.2.3.95 } indexes ● Cast data types to match destination ● … and more { user: ab123, ip: 1.2.3.XXX }

33 Confluent Hub Discover Connectors, SMTs, and converters Documentation, support, etc. Easy installation confluent.io/hub @gamussa | #SpringOne | @gprussell

34 43 Kafka Cluster Storage is Distributed 02 Scalable Store your Event Streams Reliable Durable Performant VM @gamussa | #SpringOne | @gprussell

35 Kafka scales from S to XXL Messages / sec Topics 1 10,000,000 5 250,000 25,000 500 Partitions Brokers 1,000,000 25,000 300 1,500 25 3

36 https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Store your Events as long as you want Event Streaming Paradigm Kafka = Source of Truth, stores every articleHighly since Scalable 1851 Fast (Low Latency) Durable Persistent Maintains Order Normalized assets (images, articles, bylines, etc.) @gamussa Denormalized into “Content View” | #SpringOne | @gprussell

37 Achievement Data Unlocked: All Your Data Now Available as Streams of Events @gamussa | #SpringOne | @gprussell

49 38 With Streaming SQL KSQL 03 With apps, microservices Process & Analyze your Events Streams streams or … and more Kafka consumer clients With separate frameworks … and more @gamussa | #SpringOne | @gprussell

39 Stream Processing with KSQL Process event streams to create new, continuously updated streams or tables Stream 03 Stream 01 Streaming Query Query Query Stream 02 Table CREATE TABLE OrderTotals AS SELECT * FROM … EMIT CHANGES

Up Stream Processing with KSQL com ing 40 fea tur e Query tables in Kafka from other apps, similar to a relational database Result Table Other Applications (Java, Go, Python, etc.) can directly query tables request-response via network (KSQL REST API) SELECT * FROM OrderTotals WHERE region = ‘Europe’ (KL IP8 )

41 An example use case Creating an event-driven dashboard from a currencies database Maxwell is streaming change events Results are continuously updating Elasticsearch Currencies Aggregations are computed in real-time https://github.com/garyrussell/s1p-2019

42 @StreamListener fun processCurrency(input: KStream<String, Double>) { val groupByKey: KGroupedStream<String, Double> = input.groupByKey() val countKTable = groupByKey.count() val sumKTable = groupByKey.reduce { value1, value2 !-> value1 + value2 } val avgRate = sumKTable.join(countKTable) { sum, count !-> sum / count } avgRate.toStream().to(“avg-rates”) } @gamussa | #SpringOne | @gprussell

43 @StreamListener fun processCurrency(input: KStream<String, Double>) { val groupByKey: KGroupedStream<String, Double> = input.groupByKey() val countKTable = groupByKey.count() val sumKTable = groupByKey.reduce { value1, value2 !-> value1 + value2 } val avgRate = sumKTable.join(countKTable) { sum, count !-> sum / count } avgRate.toStream().to(“avg-rates”) } @gamussa | #SpringOne | @gprussell

44 @StreamListener fun processCurrency(input: KStream<String, Double>) { val groupByKey: KGroupedStream<String, Double> = input.groupByKey() val countKTable = groupByKey.count() val sumKTable = groupByKey.reduce { value1, value2 !-> value1 + value2 } val avgRate = sumKTable.join(countKTable) { sum, count !-> sum / count } avgRate.toStream().to(“avg-rates”) } @gamussa | #SpringOne | @gprussell

45 @StreamListener fun processCurrency(input: KStream<String, Double>) { val groupByKey: KGroupedStream<String, Double> = input.groupByKey() val countKTable = groupByKey.count() val sumKTable = groupByKey.reduce { value1, value2 !-> value1 + value2 } val avgRate = sumKTable.join(countKTable) { sum, count !-> sum / count } avgRate.toStream().to(“avg-rates”) } @gamussa | #SpringOne | @gprussell

46 Where your Kafka Streams apps live KStreams Application App instance 1 network read/write fun processCurrency(input: KStream<String, Double>) { val groupByKey: KGroupedStream<String, Double> = input.groupByKey() val countKTable = groupByKey.count() val sumKTable = groupByKey.reduce { value1, value2 !-> value1 + value2 } VM val avgRate = sumKTable.join(countKTable) { sum, count !-> sum / count } avgRate.toStream().to(“avg-rates”) }

47 Where your Kafka Streams apps live KStreams Application App instance 1 … network read/write App instance n VM Elastic & Scalable Fault-tolerant Exactly-once Kafka security Aggregations Windowing Streams & Tables

48 Stream Processing with Kafka Streams apps Process event streams to create new, continuously updated streams or tables Inventory Event-driven apps and services communicate through Kafka Reporting Orders Frontend New apps can easily be added by tapping into existing event streams Shipping

49 Stream Processing with Kafka Streams apps Query your application’s tables and state from other apps Reporting App Table Result App instance 1 … App instance n request-response via network (e.g. REST API) Other Applications (Java, Go, Python, etc.) can directly query tables

50 ● Spring For Apache Kafka 2.3 What’s New? @gamussa | #SpringOne | @gprussell Producers • Option for Producer per thread • AggregatingReplyingKafkaTemplate

51 ● Spring For Apache Kafka 2.3 What’s New? @gamussa | #SpringOne | @gprussell Consumers • Consumer RecordInterceptor • Relative Seeks in ConsumerSeekAware • Configurable delay between poll()s • Micrometer Timers • Backoff between redeliveries after delivery failures • RetryingDeserializer

52 ● Spring For Apache Kafka 2.3 What’s New? @gamussa | #SpringOne | @gprussell Streams • RecoveringDeserializationExc eptionHandler • Transformers • Add headers (SpEL) • Invoke Spring Integration flows

53 ● Spring For Apache Kafka 2.3 What’s New? @gamussa | #SpringOne | @gprussell Miscellaneous • Delegating Serializer/ Deserializer

54 is a distributed event streaming platform Publish & Subscribe to Events @gamussa Store Events | #SpringOne Process & Analyze Events | @gprussell

64 55 Where to go from here for more details on event-driven architectures with Kafka @gamussa | #SpringOne | @gprussell

56 THANK YOU @gamussa viktor@confluent.io @gprussell grussell@pivotal.io cnfl.io/meetups cnfl.io/blog cnfl.io/slack