A presentation at JCON 2021 by Robin Moffatt
đ¤Building a Telegram bot with Apache Kafka and ksqlDB Robin Moffatt @rmoff
Whereâs my nearest carpark with available spaces?
How many spaces are available in this car park?
đĄTell me when a car park with spaces is available
đHow does occupancy vary over time?
Telegram @rmoff
Donât just tell me⌠show me! Demo code: https://rmoff.dev/carparks
carparks HTTP Kafka @rmoff
What are the key pieces of the design? @rmoff
Event Driven Alerts carparks HTTP Kafka @rmoff
K/V Lookups (materialised views) SELECT SPACES_AVAILABLE How many spaces are free at â FROM CARPARK WHERE NAME=âWESTGATEâ; Westgate carpark right now? ksqlDB â Kafka CARPARK_EVENTS 42 CREATE TABLE CARPARK AS SELECT LATEST(⌠GROUP BY NAME â There are 42 spaces free â @rmoff
A schema⌠carparks HTTP @rmoff
A schema⌠2020-10-14,12:28,Broadway,1132,921 2020-10-14,12:28,Kirkgate Centre,611,474 2020-10-14,12:28,Sharpe Street,98,63 𤨠?! @rmoff
My kingdom for a schema! 2020-10-14,12:28,Broadway,1132,921 âtsâ: â2020-10-14T12:28 UTC+1â, ânameâ: âBroadwayâ, âcapacityâ: 1132, âemptyâ: 921 } ⌠@rmoff
Applying a schema to streams of data source_topic ksqlDB CREATE STREAM mySource (date VARCHAR , time VARCHAR , name VARCHAR , capacity INT ) WITH (KAFKA_TOPIC=âsource_topicâ, VALUE_FORMAT=âDELIMITEDâ); Kafka @rmoff
Applying a schema to streams of data source_topic ksqlDB Kafka derived_topic CREATE STREAM mySource (date VARCHAR , time VARCHAR , name VARCHAR , capacity INT ) WITH (KAFKA_TOPIC=âsource_topicâ, VALUE_FORMAT=âDELIMITEDâ); CREATE STREAM myTargetStream WITH (VALUE_FORMAT=âPROTOBUFâ, KAFKA_TOPIC=âderived_topicâ) AS SELECT * FROM mySource; @rmoff
Integration carparks HTTP Kafka @rmoff
Streaming Integration with Kafka Connect syslog Sources Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Integration with Kafka Connect Amazon S3 Google BigQuery Sinks Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Integration with Kafka Connect Amazon S3 syslog Google BigQuery Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Analytics @rmoff
Why build it this way? @rmoff
Events @rmoff
Streams of Events @rmoff
We want to react to them as they happen @rmoff
We want to build state from a stream of events @rmoff
We want to provide the latest data in our analytics @rmoff
Apache Kafka - an Event Streaming Platform Producer Connectors Consumer The Log Connectors Streaming Engine @rmoff
Why Kafka? @rmoff
Distributed, Immutable, Event Log Old New Events are added at the end of the log @rmoff
Consumers can seek to any point Read to offset & scan Old New @rmoff
Data is not deleted once read New Old Sally is here Scan @rmoff
Consumers are independent of each other New Old Fred is here Scan Sally is here Scan @rmoff
Consumers can be added later Rick is here Scan New Old Fred is here Scan Sally is here Scan @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream Analytics @rmoff
Stream Processing with ksqlDB Source stream Applications / Microservices @rmoff
Stream Processing with ksqlDB âŚSUM(TXN_AMT) GROUP BY AC_ID AC _I D= 42 BA LA NC AC E= _I 94 D= .0 42 0 Source stream Applications / Microservices @rmoff
Under the covers of ksqlDB Photo by @rmoff on
Kafka cluster consume produce ksqlDB @rmoff
JVM Kafka cluster consume produce ksqlDB Kafka Streams RocksDB @rmoff
k & ^ Kafka Fully Managed as a Service B D l q s
Running ksqlDB - self-managed DEB, RPM, ZIP, TAR downloads http://confluent.io/download Docker images ksqlDB Server confluentinc/ksqldb-server (JVM process) âŚand many more⌠@rmoff
Why Kafka? @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Flexible, event-driven applications Event-driven alerts Key/Value lookups Streaming ETL @rmoff
on Photo by Want to learn more? CTAs, not CATs (sorry, not sorry) @rmoff
Try it out for yourself https://rmoff.dev/carparks
s e l c i t r a eep-dive D ⢠⢠⢠⢠⢠ka? f a K e ds h c n a e r p T A d s i e lat e R What . s v g min a e per r e t e S K t o n o e Z v E ut o h t i w fka a a K k f n a i K s : e t f ante KRa r a u G & ns o i t c a s n a Tr ge a r o t S & ng Processi tals n e m a d Fun ⢠⢠⢠⢠⢠e c n a m r o f r Kafka Pe a k f a K e v i t ms e t s y S Cloud-na e s ba a t a D g n Streami fka a K e h c a p ls a n Testing A r e t n I s fkaâ a K e r o l Exp ⢠⢠⢠⢠⢠Over 10 Apache K afka 101 Kafka Co nnect 10 1 Kafka Str eams 101 ksqlDB 1 01 Inside ks qlDB hours of ⢠⢠⢠⢠f ree cou rses Spring F ramewo rk and K Building afka Data Pip elines wi Event So th Kafka urcing w ith Kafka Data Me sh 101 Plus: Hands-on Quick Starts and Client Language Guides + Event Streaming Patterns + More fl developer.con uent.io
OF F2 00 $200 USD off your bill each calendar month for the rst three months when you sign up https://rmoff.dev/ccloud RM Free money! (additional $200 towards your bill đ ) Fully Managed Kafka as a Service fl fl fi
#EOF @rmoff rmoff.dev/talks youtube.com/rmoff
Further reading / watching https://rmoff.dev/kafka-talks @rmoff
Imagine youâve got a stream of data; itâs not âbig data,â but itâs certainly a lot. Within the data, youâve got some bits youâre interested in, and of those bits, youâd like to be able to query information about them at any point. Sounds fun, right? Since I mentioned âquerying,â Iâd hazard a guess that youâve got in mind an additional datastore of some sort, whether relational or NoSQL.
But what if I told youâŚthat you didnât need any datastore other than Kafka itself? What if you could ingest, filter, enrich, aggregate, and query data with just Kafka? With ksqlDB we can do just this, and I want to show you exactly how.
In this hands-on talk weâll walk through an example of building a Telegram bot in which ksqlDB provides the key/value lookups driven by a materialised view on the stream of events in Kafka. Weâll take a look at what ksqlDB is and its capabilities for processing data and driving applications, as well as integrating with other systems.
The following resources were mentioned during the presentation or are useful additional information.
Managed Apache Kafka, ksqlDB, and Schema Registry. Use code RMOFF200 when you sign up!
The pre-eminent resource for learning Apache Kafka. Free training courses, event streaming patterns, deep-dive articles, and language-specific client programming guides. Check it out!
Hereâs what was said about this presentation on social media.
