A presentation at Voxxed Days Zurich in in Zürich, Switzerland by Hans-Peter Grahsl
Taming IoT Data: Making Sense of Sensors with SQL Streaming
$ whoami ” • Hans-Peter Grahsl • working & living in Graz • technical trainer at • independent consultant & engineer • associate lecturer • @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland irregular conference speaker 2
WHAT IS STREAMING ! ❓ ! ❓ @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 3
“a type of data processing that is designed with infinite data sets in mind” — Tyler Akidau
Streaming == BIG DEAL 1. unbounded data sets are prevalent ➡ never-ending data streams need purpose-built systems 2. people crave for timely information ➡ stream processing technology aids lower latencies @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 5
BIGGEST Challenge? @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 6
These and many many more…
Today the choice is mine
Apache Kafka @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 9
STREAMING PLATFORM
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 11
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 12
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 13
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 14
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 15
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 16
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 17
Kafka’s streaming SQL engine @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 18
declarative stream processing language @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 19
skyrocketing developer productivity
unlocks streaming for the masses
KSQL’s Nature • built on top of Kafka Streams • SQL only (not embedded) • NO(!) coding skills required • extremely low entry barrier • familiar syntax and semantics • concise and expressive • joins, aggregations, windowing • UD(A)Fs and UDTFs coming soon… @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 22
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 23
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 24
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 25
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 26
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 27
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 28
KSQL Queries • per-record streaming with milliseconds latency • compiled into Kafka Streams applications • follow same execution model • distributed over multiple KSQL servers • two operation modes / deployment options: • interactive vs. headless @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 29
KSQL interactive mode • KSQL servers accessed via REST API • offers ad-hoc stream analytics • share streams & tables across users • used for exploration and during development
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 31
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 32
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 33
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 34
KSQL headless mode • streaming queries given by a SQL file • KSQL servers process SQL file • use case specific isolation • “locked-down” ➡ NO REST API access • used for production deployments
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 36
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 38
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 39
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 40
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 41
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 42
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 43
@hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 44
step 1 ingest sensor data @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 45
step 2 KSQL streaming @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 46
“You think that’s a database table you are querying now?” — Morpheus
“Instead, only try to realize the truth… there is no database table.” — Spoon Boy
step 3 connecting NoSQL @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 49
step 4 reactive notifications @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 50
step 5 live dashboards @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 51
MISSION accomplished
KSQL wrap-up • streaming with SQL … and nothing but SQL • scalable & fault-tolerant • deployable anywhere: cloud or on prem • viable for use cases of any size (XS … XXXL) • exactly-once delivery guarantee semantics @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 53
“If I have been faster it’s by streaming on the shoulders of Apache Kafka.” — my other self
Your obsession tells you to do batching. I tell you to walk away and stream with KSQL The choice is yours folks!
THANK YOU Q&A? https://bit.ly/2FaLr7w @hpgrahsl | #VDZ19 @VoxxedZurich, 19th March 2019, Switzerland 56
We are living in a data streaming era, yet until recently it has been particularly hard to leverage existing stream processing technologies. On the one hand, because dealing with data in motion has its inherent challenges. On the other hand, most frameworks and APIs which are allowing for stream processing are typically very hard to employ and/or operate - NOT so for KSQL, the newest kid in Apache Kafka’s ecosytem. Based on a simplified version of an IoT use case this session gives a gentle introduction into KSQL - Kafka’s SQL streaming engine for the masses.
Join this whirlwind tour during which we are discussing a fully-fledged streaming IoT architecture. Concretely, we are going to:
(1) ingest smart home energy data into Apache Kafka, (2) use KSQL for flexible, powerful and scalable SQL-only stream processing, (3) send raw data as well as pre-processed results to an operational NoSQL data store, (4) reactively serve data to clients in near real-time and (5) finally build informative live charts.
for free. You
can too.