A presentation at Data + AI Summit Europe by Robin Moffatt
Integrating Apache Kafka with other systems in a reliable and scalable way is often a key part of a streaming platform. Fortunately, Apache Kafka includes the Connect API that enables streaming integration both in and out of Kafka. Like any technology, understanding its architecture and deployment patterns is key to successful use, as is knowing where to go looking when things aren’t working.
This talk will discuss the key design concepts within Kafka Connect and the pros and cons of standalone vs distributed deployment modes. We’ll do a live demo of building pipelines with Kafka Connect for streaming data in from databases, and out to targets including Elasticsearch. With some gremlins along the way, we’ll go hands-on in methodically diagnosing and resolving common issues encountered with Kafka Connect. The talk will finish off by discussing more advanced topics including Single Message Transforms, and deployment of Kafka Connect in containers.
The following resources were mentioned during the presentation or are useful additional information.
Fully Managed Apache Kafka, Schema Registry, ksqlDB, and Connectors.
Blog about the options for getting data from Oracle into Kafka
A conference talk covering the options for getting data from Oracle into Kafka
Kafka 101 - introducing the concepts, the APIs, and the ecosystem
Try out the demo for yourself - all you need is Docker and Docker Compose.
Free eBooks to download, including Kafka: The Definitive Guide.
Tutorials, videos, blogs, podcasts, and more - all for developers working with Apache Kafka and Confluent Platform
Huge list of connectors for Kafka Connect
A fun blog showing what you can do with ksqlDB and Kafka
Learn all about ksqlDB in this 45 minute talk & live demo
Apache Kafka and Confluent Platform in Action! Using live streams of rail movement data in all sorts of useful ways for analysis and applications.