Streaming Data on the Shoulders of Giants

LONDON Hans-Peter Grahsl, NETCONOMY Streaming Data on the Shoulders of Giants @ hpgrahsl #MDBlocal

whoami ü working & living in Graz ! ü technical trainer at NETCONOMY ü independent engineer / consultant ü associate lecturer ü ” occasional conference speaker # #MDBLocal mongodb.com/local/london

Speed & Agility Among Top Tech Risks For businesses to stay relevant they must deliver value at a breakneck pace and be constantly seeking new sources of value…

Managing, Processing & Analyzing Data We all use Data to unlock insights and drive value #MDBLocal mongodb.com/local/london

time critical decisions preventive / predictive VALUE of data to decision-making Diminishing Value of Data near real time actionable seconds traditional business intelligence reactive minutes historical hours days months Source: Perishable Insights, Mike Gualtieri, Forrester #MDBLocal mongodb.com/local/london

Historic ETL can be rather painful • batch-driven • brittle & error prone • (too) late answers #MDBLocal Speed & Agility Antipattern mongodb.com/local/london

Alleviate some ETL pain with streaming • event-centric & stream-oriented • quick insights & fast answers #MDBLocal Speed & Agility Enabler mongodb.com/local/london

Architecture of a Modern Data Platform? Streaming Data Fabric Connected Apps Connected Apps Stream Processors Data Sources #MDBLocal Data Sinks mongodb.com/local/london

On the Shoulders of Giants MongoDB® #MDBLocal Kafka® mongodb.com/local/london

Modern Database #MDBLocal mongodb.com/local/london

Modern Database Document Model Run Anywhere Distributed & Scalable Resilient & Performant #MDBLocal mongodb.com/local/london

Apache Kafka® Minimum Viable Introduction

Streaming Platform #MDBLocal mongodb.com/local/london

Streaming Platform ü distributed ü horizontally scalable ü highly fault-tolerant #MDBLocal mongodb.com/local/london

What is Streaming? “… a type of data processing that is designed with infinite data sets in mind …” – Tyler Akidau #MDBLocal mongodb.com/local/london

“…everything that happens in a company – every customer interaction, every API request, every can be represented as real-time stream that anything else can tap into, process or react to.” database change –

“…Kafka and the whole category of stream processing represents a fundamental paradigm shift in how the digital part of a company is built, how data is used, and how applications are built. This is actually a pretty rare thing…” – Jay Kreps

KStreams App Streams API App Apps n Co Pr App Apps #MDBLocal o c du e P rA I Connect API Connect API Data Sources s e m u P rA Data Sinks I KSQL KSQL App mongodb.com/local/london

Kafka APIs in a Nutshell… • Producer & Consumer API à publish-subscribe scenarios • Connect API à streaming data integration scenarios • Streams API & KSQL à code or SQL-based streaming scenarios #MDBLocal mongodb.com/local/london

Kafka® Connect What’s it about?

Kafka Connect Basics ANY source Connect Connect ANY sink ANY à e.g. file systems, data stores, REST endpoints, … #MDBLocal mongodb.com/local/london

Kafka Connect Basics often about data stores SOURCE DB #MDBLocal Connect Connect SINK DB mongodb.com/local/london

Kafka Connect Basics or more concretely Connect Connect https://hub.confluent.io à many many more #MDBLocal mongodb.com/local/london

Kafka Connect Basics or more concretely Connect Connect https://hub.confluent.io à many many more #MDBLocal mongodb.com/local/london

Kafka® Connect How do connectors operate?

Kafka Source Connectors Source Connector S M T … S M T Converter Serialize 1…N Single Message Transforms for basic in-flight manipulations #MDBLocal mongodb.com/local/london

Kafka Sink Connectors Converter Deserialize S M T … S M T Sink Connector 1…N Single Message Transforms for basic in-flight manipulations #MDBLocal mongodb.com/local/london

Official Connector Announced at #MDBW19

MongoDB Connector for Apache Kafka Map and persist events from Kafka topics directly to MongoDB #MDBLocal Publish data changes from MongoDB into Kafka topics mongodb.com/local/london

MongoDB Connector for Apache Kafka ü developed open-source ü officially supported by MongoDB ü Verified Gold certified by Confluent Available for Download on the Confluent Hub https://www.conflent.io/hub/mongodb/kafka-connect-mongodb #MDBLocal mongodb.com/local/london

Use Cases MongoDB Connector for Apache Kafka

Single Customer View for eCommerce Source Connectors MongoDB Sinks Single Source of Truth #MDBLocal

Data Synchronization between Microservices Service 1 … Service N #MDBLocal MongoDB Sinks

Recommendation Engine for Opinion Mining Change Streams User Recommendation Engine MongoDB Source Surveys & Polls Data Change Streams #MDBLocal

Demo Scenario Let’s see it in action!

Demo Scenario REST data generation device management ! Producer API Change Streams Stream Processor #MDBLocal data serving SSE mongodb.com/local/london

Demo Scenario REST MongoDB Source Connector data generation Producer API device management MongoDB Sink Connector Change Streams Stream Processor #MDBLocal data serving SSE mongodb.com/local/london

THANK YOU!

Streaming Data on the Shoulders of Giants [NETCONOMY] Hans-Peter Grahsl https://www.surveymonkey.com/r/KXC6DB6