A presentation at GOTOpia by Robin Moffatt
Kafka has become a key data infrastructure technology, and we all have at least a vague sense that it is a messaging system, but what else is it? How can an overgrown message bus be getting this much buzz? Well, because Kafka is merely the center of a rich streaming data platform that invites detailed exploration.
In this talk, we’ll look at the entire streaming platform provided by Apache Kafka and the Confluent community components. Starting with a lonely key-value pair, we’ll build up topics, partitioning, replication, and low-level Producer and Consumer APIs. We’ll group consumers into elastically scalable, fault-tolerant application clusters, then layer on more sophisticated stream processing APIs like Kafka Streams and ksqlDB. We’ll help teams collaborate around data formats with schema management. We’ll integrate with legacy systems without writing custom code. By the time we’re done, the open-source project we thought was Big Data’s answer to message queues will have become an enterprise-grade streaming platform, all in 45 minutes.
The following resources were mentioned during the presentation or are useful additional information.
Fully Managed Apache Kafka, Schema Registry, ksqlDB, and Connectors.
Blog about the options for getting data from Oracle into Kafka
A conference talk covering the options for getting data from Oracle into Kafka
Free eBooks to download, including Kafka: The Definitive Guide.
Tutorials, videos, blogs, podcasts, and more - all for developers working with Apache Kafka and Confluent Platform
Huge list of connectors for Kafka Connect
A fun blog showing what you can do with ksqlDB and Kafka
Learn all about ksqlDB in this 45 minute talk & live demo
Learn all about Kafka Connect (including the connectors available with ksqlDB)
Apache Kafka and Confluent Platform in Action! Using live streams of rail movement data in all sorts of useful ways for analysis and applications.
Here’s what was said about this presentation on Twitter.