A presentation at Big Mountain Data and Dev Conference by Robin Moffatt
đ€Building a Telegram bot with Apache Kafka and ksqlDB @rmoff Robin Moffatt
Whereâs my nearest carpark with available spaces?
How many spaces are available in this car park?
đĄTell me when a car park with spaces is available
đHow does occupancy vary over time?
$ whoami > Robin Moffatt (@rmoff) > Senior Developer Advocate at Confluent (Apache Kafka, not Wikis đ) > Working in data & analytics since 2001 > Oracle ACE Director (Alumnus) http://rmoff.dev/talks · http://rmoff.dev/blog · http://rmoff.dev/youtube @rmoff
Telegram @rmoff
Donât just tell me⊠show me! Demo code: https://rmoff.dev/carparks
carparks HTTP Kafka @rmoff
What are the key pieces of the design? @rmoff
Event Driven Alerts carparks HTTP Kafka @rmoff
K/V Lookups (materialised views) SELECT SPACES_AVAILABLE How many spaces are free at â FROM CARPARK WHERE NAME=âWESTGATEâ; Westgate carpark right now? ksqlDB â Kafka CARPARK_EVENTS 42 CREATE TABLE CARPARK AS SELECT LATEST(⊠GROUP BY NAME â There are 42 spaces free â @rmoff
A schema⊠carparks HTTP @rmoff
A schema⊠2020-10-14,12:28,Broadway,1132,921 2020-10-14,12:28,Kirkgate Centre,611,474 2020-10-14,12:28,Sharpe Street,98,63 ?! @rmoff
My kingdom for a schema! 2020-10-14,12:28,Broadway,1132,921 2020-10-14,12:28,Kirkgate Centre,611,474 2020-10-14,12:28,Sharpe Street,98,63 đ { âtsâ: â2020-10-14T12:28 UTC+1â, ânameâ: âBroadwayâ, âcapacityâ: 1132, âemptyâ: 921 } ⊠@rmoff
Applying a schema to streams of data source_topic ksqlDB CREATE STREAM mySource (date VARCHAR , time VARCHAR , name VARCHAR , capacity INT ) WITH (KAFKA_TOPIC=âsource_topicâ, VALUE_FORMAT=âDELIMITEDâ); Kafka @rmoff
Applying a schema to streams of data source_topic ksqlDB Kafka derived_topic CREATE STREAM mySource (date VARCHAR , time VARCHAR , name VARCHAR , capacity INT ) WITH (KAFKA_TOPIC=âsource_topicâ, VALUE_FORMAT=âDELIMITEDâ); CREATE STREAM myTargetStream WITH (VALUE_FORMAT=âPROTOBUFâ, KAFKA_TOPIC=âderived_topicâ) AS SELECT * FROM mySource; @rmoff
Integration carparks HTTP Kafka @rmoff
Streaming Integration with Kafka Connect syslog Sources Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Integration with Kafka Connect Amazon S3 Google BigQuery Sinks Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Integration with Kafka Connect Amazon S3 syslog Google BigQuery Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Analytics @rmoff
Why build it this way? @rmoff
Events @rmoff
Streams of Events @rmoff
We want to react to them as they happen @rmoff
We want to build state from a stream of events @rmoff
We want to provide the latest data in our analytics @rmoff
Apache Kafka - an Event Streaming Platform Producer Connectors Consumer The Log Connectors Streaming Engine @rmoff
Why Kafka? @rmoff
Distributed, Immutable, Event Log Old New Events are added at the end of the log @rmoff
Consumers can seek to any point Read to offset & scan Old New @rmoff
Data is not deleted once read New Old Sally is here Scan @rmoff
Consumers are independent of each other New Old Fred is here Scan Sally is here Scan @rmoff
Consumers can be added later Rick is here Scan New Old Fred is here Scan Sally is here Scan @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream Analytics @rmoff
Stream Processing with ksqlDB Source stream Applications / Microservices @rmoff
Stream Processing with ksqlDB âŠSUM(TXN_AMT) GROUP BY AC_ID AC _I D= 42 BA LA NC AC E= _I 94 D= .0 42 0 Source stream Applications / Microservices @rmoff
Under the covers of ksqlDB Photo by @rmoff on
Kafka cluster consume produce ksqlDB @rmoff
JVM Kafka cluster consume produce ksqlDB Kafka Streams RocksDB @rmoff
k & ^ Kafka Fully Managed as a Service B D l q s
Running ksqlDB - self-managed DEB, RPM, ZIP, TAR downloads http://confluent.io/download Docker images ksqlDB Server confluentinc/ksqldb-server (JVM process) âŠand many more⊠@rmoff
Why Kafka? @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Flexible, event-driven applications Event-driven alerts Key/Value lookups Streaming ETL @rmoff
on Photo by Want to learn more? CTAs, not CATs (sorry, not sorry) @rmoff
Try it out for yourself https://rmoff.dev/carparks
60 DE VA DV $200 USD off your bill each calendar month for the first three months when you sign up https://rmoff.dev/ccloud Free money! (additional $60 towards your bill đ ) Fully Managed Kafka as a Service * T&C: https://www.confluent.io/confluent-cloud-promo-disclaimer
Learn Kafka. Start building with Apache Kafka at Confluent Developer. developer.confluent.io
Confluent Community Slack group cnfl.io/slack @rmoff
Further reading / watching https://rmoff.dev/kafka-talks @rmoff
#EOF https://talks.rmoff.net @rmoff
Imagine youâve got a stream of data; itâs not âbig data,â but itâs certainly a lot. Within the data, youâve got some bits youâre interested in, and of those bits, youâd like to be able to query information about them at any point. Sounds fun, right? Since I mentioned âquerying,â Iâd hazard a guess that youâve got in mind an additional datastore of some sort, whether relational or NoSQL. But what if I told youâŠthat you didnât need any datastore other than Kafka itself? What if you could ingest, filter, enrich, aggregate, and query data with just Kafka? With ksqlDB we can do just this, and I want to show you exactly how. In this hands-on talk weâll walk through an example of building a Telegram bot in which ksqlDB provides the key/value lookups driven by a materialised view on the stream of events in Kafka. Weâll take a look at what ksqlDB is and its capabilities for processing data and driving applications, as well as integrating with other systems.
The following resources were mentioned during the presentation or are useful additional information.
Fully Managed Apache Kafka, Schema Registry, ksqlDB, and Connectors.
Blog about the options for getting data from Oracle into Kafka
A conference talk covering the options for getting data from Oracle into Kafka
Kafka 101 - introducing the concepts, the APIs, and the ecosystem
Try out the demo for yourself - all you need is Docker and Docker Compose.
Free eBooks to download, including Kafka: The Definitive Guide.
Tutorials, videos, blogs, podcasts, and more - all for developers working with Apache Kafka and Confluent Platform
Huge list of connectors for Kafka Connect
A fun blog showing what you can do with ksqlDB and Kafka
Learn all about ksqlDB in this 45 minute talk & live demo
Learn all about Kafka Connect (including the connectors available with ksqlDB)
Apache Kafka and Confluent Platform in Action! Using live streams of rail movement data in all sorts of useful ways for analysis and applications.
Hereâs what was said about this presentation on social media.
