Advanced Change Data Streaming Patterns in Distributed Systems

A presentation at Kafka Summit Europe 2021 in May 2021 in by Hans-Peter Grahsl

Slide 1

Kafka Summit Europe 2021 Advanced Change Data Streaming Patterns in Distributed Systems Gunnar Morling Software Engineer, Red Hat @gunnarmorling Hans-Peter Grahsl Technical Trainer, Netconomy @hpgrahsl

Slide 2

Gunnar Morling ● Open source software engineer at Red Hat ○ Debezium ○ Quarkus ● Spec Lead for Bean Validation 2.0 ● Java Champion ● @gunnarmorling #CDCPatterns @gunnarmorling @hpgrahsl

Slide 3

Hans-Peter Grahsl ● Technical Trainer at NETCONOMY ● Independent Engineer & Consultant ● Confluent Community Catalyst ● MongoDB Champion ● @hpgrahsl #CDCPatterns @gunnarmorling @hpgrahsl

Slide 4

Today’s Objective: Learn About… … implemented using Change Data Capture

Slide 5

Debezium — Log-based Change Data Capture ● Taps into TX log to capture INSERT/UPDATE/DELETE events ● Propagated to consumers via Apache Kafka and Kafka Connect #CDCPatterns @gunnarmorling @hpgrahsl

Slide 6

Outbox Pattern

Slide 7

The Problem: Microservices Data Exchange ● Services need to update their database, ● send messages to other services, ● and that consistently! #CDCPatterns @gunnarmorling @hpgrahsl

Slide 8

Outbox Pattern “Dual writes” are prone to inconsistencies! #CDCPatterns @gunnarmorling @hpgrahsl

Slide 9

Outbox Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 10

Outbox Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 11

Outbox Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 12

Outbox Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 13

Strangler Fig Pattern

Slide 14

The Problem: Migrating Systems ● Gradually evolve from old into new ● Support temporary coexistence ● Avoid big bang cut-over #CDCPatterns @gunnarmorling @hpgrahsl

Slide 15

Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 16

Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 17

Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 18

Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 19

Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 20

Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 21

Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 22

Benefits ● Incremental migration → “baby steps” ● Pause or stop migration without losing spent efforts ● Migration steps ideally reversible Rationale: ⚠ minimize risk ⚠ #CDCPatterns @gunnarmorling @hpgrahsl

Slide 23

CDC Close-Up #CDCPatterns @gunnarmorling @hpgrahsl

Slide 24

Enhanced CDC Processing #CDCPatterns @gunnarmorling @hpgrahsl

Slide 25

Enhanced CDC Processing #CDCPatterns @gunnarmorling @hpgrahsl

Slide 26

Enhanced CDC Processing #CDCPatterns @gunnarmorling @hpgrahsl

Slide 27

Saga Pattern

Slide 28

The Problem: Long-running Business Transactions ● Multiple services need to act collaboratively to achieve a consistent outcome ● Without 2-phase commit protocols ● Ensure correctness in case of failures #CDCPatterns @gunnarmorling @hpgrahsl

Slide 29

Saga Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 30

Saga Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 31

Saga Pattern #CDCPatterns @gunnarmorling @hpgrahsl

Slide 32

Demo

Slide 33

Wrap-Up

Slide 34

Takeaways ● CDC: a powerful tool in the box for event-driven architectures ● Debezium: open-source CDC for a variety of databases ● Call to Action: Would you like built-in Saga support? #CDCPatterns @gunnarmorling @hpgrahsl

Slide 35

Resources ● Outbox implementation https://debezium.io/blog/2019/02/19/reliable-microservices-data -exchange-with-the-outbox-pattern/ ● Strangler fig pattern https://martinfowler.com/bliki/StranglerFigApplication.html ● Saga implementation https://www.infoq.com/articles/saga-orchestration-outbox/ #CDCPatterns @gunnarmorling @hpgrahsl

Slide 36

Q&A 📧 gunnar@hibernate.org @gunnarmorling grahslhp@gmail.com @hpgrahsl 📧 #CDCPatterns @gunnarmorling @hpgrahsl

Slide 37

Image Credits In Order of Appearance Unsplash https://unsplash.com/license © Shiro hatori https://unsplash.com/photos/WR-ifjFy4CI © Pablo García Saldaña https://unsplash.com/photos/lPQIndZz8Mo © David Clode https://unsplash.com/photos/T49WTav4LgU © Aaron Burden https://unsplash.com/photos/GFpxQ2ZyNc0 © Nathan Dumlao https://unsplash.com/photos/wQDysNUCKfw © mari lezhava https://unsplash.com/photos/q65bNe9fW-w © Michał Parzuchowski https://unsplash.com/photos/Bt0PM7cNJFQ © Charles Forerunner https://unsplash.com/photos/3fPXt37X6UQ Flickr Attribution 2.0 Generic https://creativecommons.org/licenses/by/2.0/ © Thomas Kamann https://flic.kr/p/coa2c CC0 1.0 Universal Public Domain Dedication https://creativecommons.org/publicdomain/zero/1.0/ © Wall Boat https://flic.kr/p/Y6zkmX Attribution-ShareAlike 2.0 Generic https://creativecommons.org/licenses/by-sa/2.0/ © Maria Eklind https://flic.kr/p/251HmVw