A presentation at Kafka Summit Europe 2021 by Hans-Peter Grahsl
Kafka Summit Europe 2021 Advanced Change Data Streaming Patterns in Distributed Systems Gunnar Morling Software Engineer, Red Hat @gunnarmorling Hans-Peter Grahsl Technical Trainer, Netconomy @hpgrahsl
Gunnar Morling ● Open source software engineer at Red Hat ○ Debezium ○ Quarkus ● Spec Lead for Bean Validation 2.0 ● Java Champion ● @gunnarmorling #CDCPatterns @gunnarmorling @hpgrahsl
Hans-Peter Grahsl ● Technical Trainer at NETCONOMY ● Independent Engineer & Consultant ● Confluent Community Catalyst ● MongoDB Champion ● @hpgrahsl #CDCPatterns @gunnarmorling @hpgrahsl
Today’s Objective: Learn About… … implemented using Change Data Capture
Debezium — Log-based Change Data Capture ● Taps into TX log to capture INSERT/UPDATE/DELETE events ● Propagated to consumers via Apache Kafka and Kafka Connect #CDCPatterns @gunnarmorling @hpgrahsl
Outbox Pattern
The Problem: Microservices Data Exchange ● Services need to update their database, ● send messages to other services, ● and that consistently! #CDCPatterns @gunnarmorling @hpgrahsl
Outbox Pattern “Dual writes” are prone to inconsistencies! #CDCPatterns @gunnarmorling @hpgrahsl
Outbox Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Outbox Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Outbox Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Outbox Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Strangler Fig Pattern
The Problem: Migrating Systems ● Gradually evolve from old into new ● Support temporary coexistence ● Avoid big bang cut-over #CDCPatterns @gunnarmorling @hpgrahsl
Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Strangler Fig Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Benefits ● Incremental migration → “baby steps” ● Pause or stop migration without losing spent efforts ● Migration steps ideally reversible Rationale: ⚠ minimize risk ⚠ #CDCPatterns @gunnarmorling @hpgrahsl
CDC Close-Up #CDCPatterns @gunnarmorling @hpgrahsl
Enhanced CDC Processing #CDCPatterns @gunnarmorling @hpgrahsl
Enhanced CDC Processing #CDCPatterns @gunnarmorling @hpgrahsl
Enhanced CDC Processing #CDCPatterns @gunnarmorling @hpgrahsl
Saga Pattern
The Problem: Long-running Business Transactions ● Multiple services need to act collaboratively to achieve a consistent outcome ● Without 2-phase commit protocols ● Ensure correctness in case of failures #CDCPatterns @gunnarmorling @hpgrahsl
Saga Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Saga Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Saga Pattern #CDCPatterns @gunnarmorling @hpgrahsl
Demo
Wrap-Up
Takeaways ● CDC: a powerful tool in the box for event-driven architectures ● Debezium: open-source CDC for a variety of databases ● Call to Action: Would you like built-in Saga support? #CDCPatterns @gunnarmorling @hpgrahsl
Resources ● Outbox implementation https://debezium.io/blog/2019/02/19/reliable-microservices-data -exchange-with-the-outbox-pattern/ ● Strangler fig pattern https://martinfowler.com/bliki/StranglerFigApplication.html ● Saga implementation https://www.infoq.com/articles/saga-orchestration-outbox/ #CDCPatterns @gunnarmorling @hpgrahsl
Q&A 📧 gunnar@hibernate.org @gunnarmorling grahslhp@gmail.com @hpgrahsl 📧 #CDCPatterns @gunnarmorling @hpgrahsl
Image Credits In Order of Appearance Unsplash https://unsplash.com/license © Shiro hatori https://unsplash.com/photos/WR-ifjFy4CI © Pablo García Saldaña https://unsplash.com/photos/lPQIndZz8Mo © David Clode https://unsplash.com/photos/T49WTav4LgU © Aaron Burden https://unsplash.com/photos/GFpxQ2ZyNc0 © Nathan Dumlao https://unsplash.com/photos/wQDysNUCKfw © mari lezhava https://unsplash.com/photos/q65bNe9fW-w © Michał Parzuchowski https://unsplash.com/photos/Bt0PM7cNJFQ © Charles Forerunner https://unsplash.com/photos/3fPXt37X6UQ Flickr Attribution 2.0 Generic https://creativecommons.org/licenses/by/2.0/ © Thomas Kamann https://flic.kr/p/coa2c CC0 1.0 Universal Public Domain Dedication https://creativecommons.org/publicdomain/zero/1.0/ © Wall Boat https://flic.kr/p/Y6zkmX Attribution-ShareAlike 2.0 Generic https://creativecommons.org/licenses/by-sa/2.0/ © Maria Eklind https://flic.kr/p/251HmVw
Microservices are one of the big trends in software engineering of the last few years; organising business functionality in several self-contained, loosely coupled services helps teams to work efficiently, make the most suitable technical decisions, and react quickly to new business requirements.
In this session we’ll discuss and showcase how open-source change data capture (CDC) with Debezium can help developers with typical challenges they often face when working on microservices. Come and join us to learn how to:
The following resources were mentioned during the presentation or are useful additional information.
NOTE: registration at event platform https://kafkasummit.io required!
Here’s what was said about this presentation on social media.
for free. You
can too.