Kafka as a Platform: the Ecosystem from the Ground Up

Kafka as a Platform: the Ecosystem from the Ground Up Robin Moffatt @rmoff

EVENTS @rmoff

EVENTS @rmoff

• • EVENTS d e n e p p a h g n i h t e Som d e n e p p a h t a Wh

Human generated events A Sale A Stock movement @rmoff

Machine generated events Networking IoT Applications @rmoff

EVENTS are EVERYWHERE @rmoff

EVENTS y r e v ^ are POWERFUL @rmoff

K V

LOG @rmoff

K V

K V

K V

K V

K V

K V

K V

Immutable Event Log Old New Events are added at the end of the log @rmoff

TOPICS @rmoff

Topics Clicks Orders Customers Topics are similar in concept to tables in a database @rmoff

PARTITIONS @rmoff

Partitions Clicks p0 P1 P2 Messages are guaranteed to be strictly ordered within a partition @rmoff

PUB / SUB @rmoff

PUB / SUB @rmoff

Producing data Old New Messages are added at the end of the log @rmoff

partition 0 … partition 1 producer … partition 2 … Partitioned Topic

package main import ( “gopkg.in/confluentinc/confluent-kafka-go.v1/kafka” ) func main() { topic := “test_topic” p, _ := kafka.NewProducer(&kafka.ConfigMap{ “bootstrap.servers”: “localhost:9092”}) defer p.Close() p.Produce(&kafka.Message{ TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: 0}, Value: []byte(“Hello world”)}, nil) }

Producing to Kafka - No Key Time Partition 1 Partition 2 Partition 3 Messages will be produced in a round robin fashion Partition 4 @rmoff

Producing to Kafka - With Key Time Partition 1 A Partition 2 B hash(key) % numPartitions = N Partition 3 C Partition 4 D @rmoff

Producers partition 0 … partition 1 producer … partition 2 … Partitioned Topic • A client application • Puts messages into topics • Handles partitioning, network protocol • Java, Go, .NET, C/C++, Python • Also every other language Plus REST proxy if not

PUB / SUB @rmoff

Consuming data - access is only sequential Read to offset & scan Old New @rmoff

Consumers have a position of their own Old New Sally is here Scan @rmoff

Consumers have a position of their own Old New Fred is here Scan Sally is here Scan @rmoff

Consumers have a position of their own Rick is here Scan Old New Fred is here Scan Sally is here Scan @rmoff

c, _ := kafka.NewConsumer(&cm) defer c.Close() c.Subscribe(topic, nil) for { select { case ev := <-c.Events(): switch ev.(type) { case *kafka.Message: km := ev.(*kafka.Message) fmt.Printf(“✅ Message ‘%v’ received from topic ‘%v’\n”, string(km.Value), string(*km.TopicPartition.Topic)) } } }

partition 0 … partition 1 … partition 2 … Partitioned Topic consumer A

Consuming From Kafka - Multiple Consumers Partition 1 A1 Partition 2 Partition 3 A2 Partition 4 @rmoff

Consuming From Kafka - Grouped Consumers Partition 1 Partition 2 Partition 3 CC1 1 CA1 1 A2 Partition 4 @rmoff

Consuming From Kafka - Grouped Consumers Partition 1 Partition 2 Partition 3 C1 C2 C3 C4 Partition 4 @rmoff

Consuming From Kafka - Grouped Consumers Partition 1 Partition 2 Partition 3 C1 C2 C3 3 Partition 4 @rmoff

Consuming From Kafka - Grouped Consumers Partition 1 Partition 2 Partition 3 C1 C2 C3 Partition 4 @rmoff

Consumers partition 0 … partition 1 … consumer A consumer A consumer A partition 2 … Partitioned Topic consumer B • A client application • Reads messages from topics • Horizontally, elastically scalable (if stateless) • Java, Go, .NET, C/C++, Python, everything else Plus REST proxy if not

BROKERS and REPLICATION @rmoff

Leader Partition Leadership and Replication Follower Partition 1 Partition 2 Partition 3 Partition 4 Broker 1 Broker 2 Broker 3 @rmoff

Leader Partition Leadership and Replication Follower Partition 1 Partition 1 Partition 1 Partition 2 Partition 2 Partition 2 Partition 3 Partition 3 Partition 3 Partition 4 Partition 4 Partition 4 Broker 1 Broker 2 Broker 3 @rmoff

Leader Partition Leadership and Replication Follower Partition 1 Partition 1 Partition 1 Partition 2 Partition 2 Partition 2 Partition 3 Partition 3 Partition 3 Partition 4 Partition 4 Partition 4 Broker 1 Broker 2 Broker 3 @rmoff

So far, this is Pretty good @rmoff

So far, this is Pretty good but I’ve not finished yet… @rmoff

Streaming Pipelines Amazon S3 RDBMS HDFS @rmoff

Evolve processing from old systems to new Existing App New App <x> RDBMS @rmoff

Streaming Integration with Kafka Connect syslog Sources Kafka Connect Kafka Brokers @rmoff

Streaming Integration with Kafka Connect Amazon Sinks Google Kafka Connect Kafka Brokers @rmoff

Streaming Integration with Kafka Connect Amazon syslog Google Kafka Connect Kafka Brokers @rmoff

Look Ma, No Code! { “connector.class”: “io.confluent.connect.jdbc.JdbcSourceConnector”, “connection.url”: “jdbc:mysql://asgard:3306/demo”, “table.whitelist”: “sales,orders,customers” } @rmoff

Extensible Connector Transform(s) Converter @rmoff

hub.confluent.io @rmoff

K V

K V

K V

K V

K V ? s i h t s ’ t a h w … t i a W

Lack of schemas – Coupling teams and services 2001 2001 Citrus Heights-Sunrise Blvd Citrus_Hghts 60670001 3400293 34 SAC Sacramento SV Sacramento Valley SAC Sacramento County APCD SMA8 Sacramento Metropolitan Area CA 6920 Sacramento 28 6920 13588 7400 Sunrise Blvd 95610 38 41 56 38.6988889 121 16 15.98999977 -121.271111 10 4284781 650345 52 @rmoff

Serialisation & Schemas JSON Avro Protobuf Schema JSON CSV @rmoff

Serialisation & Schemas JSON Avro Protobuf Schema JSON CSV 👍 👍 👍 😬 https://rmoff.dev/qcon-schemas @rmoff

Schemas Schema Registry Topic producer … consumer

partition 0 … partition 1 … consumer A consumer A consumer A partition 2 … consumer B Partitioned Topic @rmoff

consumer A consumer A consumer A @rmoff

{ @rmoff

.stream(“widgets”, Consumed.with(stringSerde, widgetsSerde)) .filter( (key, widget) -> widget.getColour().equals(“RED”) ) .to(“widgets_red”, Produced.with(stringSerde, widgetsSerde));

Streams Application Streams Application Streams Application @rmoff

Stream Processing with ksqlDB Stream: widgets ksqlDB CREATE STREAM widgets_red AS SELECT * FROM widgets WHERE colour=’RED’; Stream: widgets_red @rmoff

{ @rmoff

FROM WIDGETS WHERE WEIGHT_G > 120 { SELECT COUNT(*) FROM WIDGETS GROUP BY PRODUCTION_LINE SELECT AVG(TEMP_CELCIUS) AS TEMP FROM WIDGETS GROUP BY SENSOR_ID HAVING TEMP>20 ‘connector.class’ = ‘S3Connector’, ‘topics’ = ‘widgets’ …);

Summary @rmoff

@rmoff

K V @rmoff

K V @rmoff

The Log @rmoff

Producer Consumer The Log @rmoff

Producer Consumer The Log Connectors @rmoff

Producer Consumer The Log Connectors Streaming Engine @rmoff

Apache Kafka Producer Consumer The Log Connectors Streaming Engine @rmoff

Producer Security Schema Registry Consumer The Log Streaming Engine ksqlDB REST Proxy Connectors Confluent Control Center

s e l c i t r a eep-dive D • • • • • ka? f a K e ds h c n a e r p T A d s i e lat e R What . s v g min a e per r e t e S K t o n o e Z v E ut o h t i w fka a a K k f n a i K s : e t f ante KRa r a u G & ns o i t c a s n a Tr ge a r o t S & ng Processi tals n e m a d Fun • • • • • e c n a m r o f r Kafka Pe a k f a K e v i t ms e t s y S Cloud-na e s ba a t a D g n Streami fka a K e h c a p ls a n Testing A r e t n I s fka’ a K e r o l Exp • • • • • Over 10 Apache K afka 101 Kafka Co nnect 10 1 Kafka Str eams 101 ksqlDB 1 01 Inside ks qlDB hours of • • • • f ree cou rses Spring F ramewo rk and K Building afka Data Pip elines wi Event So th Kafka urcing w ith Kafka Data Me sh 101 Plus: Hands-on Quick Starts and Client Language Guides + Event Streaming Patterns + More fl developer.con uent.io

#EOF @rmoff rmoff.dev/talks youtube.com/rmoff