Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!

@rmoff / http://cnfl.io/ksql Kafka → Elasticsearch curl -X “POST” “http://kafka-connect-cp:18083/connectors/” \ -H “Content-Type: application/json” \ -d ‘{ “name”: “es_sink_lisa18”, “config”: { “connector.class”: “io.confluent.connect.elasticsearch.ElasticsearchSinkConnector”, “key.converter”: “org.apache.kafka.connect.storage.StringConverter”, “value.converter”: “org.apache.kafka.connect.json.JsonConverter”, “value.converter.schemas.enable”: false, “topics”: “lisa18”, “key.ignore”: “true”, “schema.ignore”: “true”, “type.name”: “type.name=kafkaconnect”, “connection.url”: “http://elasticsearch:9200” } }’ Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!

@rmoff / http://cnfl.io/ksql { “rating_id”: 5313, “user_id”: 3, “stars”: 4, “route_id”: 6975, “rating_time”: 1519304105213, “channel”: “web”, “message”: “worst. flight. ever. #neveragain” } Producer API t c e n n o C a k f a K Join each rating to customer data RATINGS_WITH_CUSTOMER_DATA Filter for just PLATINUM customers UNHAPPY_PLATINUM_CUSTOMERS { “id”: 3, “first_name”: “Merilyn”, “last_name”: “Doughartie”, “email”: “mdoughartie1@dedecms.com”, “gender”: “Female”, “club_status”: “platinum”, “comments”: “none” CREATE STREAM UNHAPPY_PLATINUM_CUSTOMERS AS SELECT * FROM RATINGS_WITH_CUSTOMER_DATA WHERE STARS < 3 } Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!

Confluent Open Source : Apache Kafka with a bunch of cool stuff! For free! @rmoff / http://cnfl.io/ksql Log Events Database Changes loT Data Web Events … Confluent Platform Data Integration Real-time Applications Monitoring & Administration Confluent Control Center | Security Confluent Platform Transformations Hadoop Operations Replicator | Auto Data Balancing Custom Apps Database Data Compatibility Schema Registry SQL Stream Processing KSQL Data Warehouse Development and Connectivity Clients | Connectors | REST Proxy | CLI CRM Monitoring Apache Kafka® Core | Connect API | Streams API … CUSTOMER SELF-MANAGED Datacenter Analytics … CONFLUENT FULLY-MANAGED Public Cloud Confluent Cloud Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!