🤖 Building a Telegram bot with Apache Kafka and ksqlDB

🤖Building a Telegram bot with Apache Kafka and ksqlDB Robin Moffatt @rmoff

Where’s my nearest carpark with available spaces?

How many spaces are available in this car park?

💡Tell me when a car park with spaces is available

📈How does occupancy vary over time?

Telegram @rmoff

Don’t just tell me… show me! Demo code: https://rmoff.dev/carparks

carparks HTTP Kafka @rmoff

What are the key pieces of the design? @rmoff

Event Driven Alerts carparks HTTP Kafka @rmoff

K/V Lookups (materialised views) SELECT SPACES_AVAILABLE How many spaces are free at “ FROM CARPARK WHERE NAME=’WESTGATE’; Westgate carpark right now? ksqlDB ” Kafka CARPARK_EVENTS 42 CREATE TABLE CARPARK AS SELECT LATEST(… GROUP BY NAME “ There are 42 spaces free ” @rmoff

A schema… carparks HTTP @rmoff

A schema… 2020-10-14,12:28,Broadway,1132,921 2020-10-14,12:28,Kirkgate Centre,611,474 2020-10-14,12:28,Sharpe Street,98,63 🤨 ?! @rmoff

My kingdom for a schema! 2020-10-14,12:28,Broadway,1132,921 “ts”: “2020-10-14T12:28 UTC+1”, “name”: “Broadway”, “capacity”: 1132, “empty”: 921 } … @rmoff

Applying a schema to streams of data source_topic ksqlDB CREATE STREAM mySource (date VARCHAR , time VARCHAR , name VARCHAR , capacity INT ) WITH (KAFKA_TOPIC=’source_topic’, VALUE_FORMAT=’DELIMITED’); Kafka @rmoff

Applying a schema to streams of data source_topic ksqlDB Kafka derived_topic CREATE STREAM mySource (date VARCHAR , time VARCHAR , name VARCHAR , capacity INT ) WITH (KAFKA_TOPIC=’source_topic’, VALUE_FORMAT=’DELIMITED’); CREATE STREAM myTargetStream WITH (VALUE_FORMAT=’PROTOBUF’, KAFKA_TOPIC=’derived_topic’) AS SELECT * FROM mySource; @rmoff

Integration carparks HTTP Kafka @rmoff

Streaming Integration with Kafka Connect syslog Sources Tasks Workers Kafka Connect Kafka Brokers @rmoff

Streaming Integration with Kafka Connect Amazon S3 Google BigQuery Sinks Tasks Workers Kafka Connect Kafka Brokers @rmoff

Streaming Integration with Kafka Connect Amazon S3 syslog Google BigQuery Tasks Workers Kafka Connect Kafka Brokers @rmoff

Streaming Analytics @rmoff

Why build it this way? @rmoff

Events @rmoff

Streams of Events @rmoff

We want to react to them as they happen @rmoff

We want to build state from a stream of events @rmoff

We want to provide the latest data in our analytics @rmoff

Apache Kafka - an Event Streaming Platform Producer Connectors Consumer The Log Connectors Streaming Engine @rmoff

Why Kafka? @rmoff

Distributed, Immutable, Event Log Old New Events are added at the end of the log @rmoff

Consumers can seek to any point Read to offset & scan Old New @rmoff

Data is not deleted once read New Old Sally is here Scan @rmoff

Consumers are independent of each other New Old Fred is here Scan Sally is here Scan @rmoff

Consumers can be added later Rick is here Scan New Old Fred is here Scan Sally is here Scan @rmoff

Stream Processing with ksqlDB Source stream @rmoff

Stream Processing with ksqlDB Source stream @rmoff

Stream Processing with ksqlDB Source stream @rmoff

Stream Processing with ksqlDB Source stream Analytics @rmoff

Stream Processing with ksqlDB Source stream Applications / Microservices @rmoff

Stream Processing with ksqlDB …SUM(TXN_AMT) GROUP BY AC_ID AC _I D= 42 BA LA NC AC E= _I 94 D= .0 42 0 Source stream Applications / Microservices @rmoff

Under the covers of ksqlDB Photo by @rmoff on

Kafka cluster consume produce ksqlDB @rmoff

JVM Kafka cluster consume produce ksqlDB Kafka Streams RocksDB @rmoff

k & ^ Kafka Fully Managed as a Service B D l q s

Running ksqlDB - self-managed DEB, RPM, ZIP, TAR downloads http://confluent.io/download Docker images ksqlDB Server confluentinc/ksqldb-server (JVM process) …and many more… @rmoff

Why Kafka? @rmoff

Stream Store Process Integrate @rmoff

Stream Store Process Integrate @rmoff

Stream Store Process Integrate @rmoff

Stream Store Process Integrate @rmoff

Stream Store Process Integrate @rmoff

Flexible, event-driven applications Event-driven alerts Key/Value lookups Streaming ETL @rmoff

on Photo by Want to learn more? CTAs, not CATs (sorry, not sorry) @rmoff

Try it out for yourself https://rmoff.dev/carparks

s e l c i t r a eep-dive D • • • • • ka? f a K e ds h c n a e r p T A d s i e lat e R What . s v g min a e per r e t e S K t o n o e Z v E ut o h t i w fka a a K k f n a i K s : e t f ante KRa r a u G & ns o i t c a s n a Tr ge a r o t S & ng Processi tals n e m a d Fun • • • • • e c n a m r o f r Kafka Pe a k f a K e v i t ms e t s y S Cloud-na e s ba a t a D g n Streami fka a K e h c a p ls a n Testing A r e t n I s fka’ a K e r o l Exp • • • • • Over 10 Apache K afka 101 Kafka Co nnect 10 1 Kafka Str eams 101 ksqlDB 1 01 Inside ks qlDB hours of • • • • f ree cou rses Spring F ramewo rk and K Building afka Data Pip elines wi Event So th Kafka urcing w ith Kafka Data Me sh 101 Plus: Hands-on Quick Starts and Client Language Guides + Event Streaming Patterns + More fl developer.con uent.io

OF F2 00 $200 USD off your bill each calendar month for the rst three months when you sign up https://rmoff.dev/ccloud RM Free money! (additional $200 towards your bill 😄 ) Fully Managed Kafka as a Service fl fl fi

• T&C: https://www.con uent.io/con uent-cloud-promo-disclaimer

#EOF @rmoff rmoff.dev/talks youtube.com/rmoff

Further reading / watching https://rmoff.dev/kafka-talks @rmoff