A presentation at MongoDB.local London 2019 in in London, UK by Hans-Peter Grahsl
LONDON Hans-Peter Grahsl, NETCONOMY Streaming Data on the Shoulders of Giants @ hpgrahsl #MDBlocal
whoami ü working & living in Graz ! ü technical trainer at NETCONOMY ü independent engineer / consultant ü associate lecturer ü ” occasional conference speaker # #MDBLocal mongodb.com/local/london
Speed & Agility Among Top Tech Risks For businesses to stay relevant they must deliver value at a breakneck pace and be constantly seeking new sources of value…
Managing, Processing & Analyzing Data We all use Data to unlock insights and drive value #MDBLocal mongodb.com/local/london
time critical decisions preventive / predictive VALUE of data to decision-making Diminishing Value of Data near real time actionable seconds traditional business intelligence reactive minutes historical hours days months Source: Perishable Insights, Mike Gualtieri, Forrester #MDBLocal mongodb.com/local/london
Historic ETL can be rather painful • batch-driven • brittle & error prone • (too) late answers #MDBLocal Speed & Agility Antipattern mongodb.com/local/london
Alleviate some ETL pain with streaming • event-centric & stream-oriented • quick insights & fast answers #MDBLocal Speed & Agility Enabler mongodb.com/local/london
Architecture of a Modern Data Platform? Streaming Data Fabric Connected Apps Connected Apps Stream Processors Data Sources #MDBLocal Data Sinks mongodb.com/local/london
On the Shoulders of Giants MongoDB® #MDBLocal Kafka® mongodb.com/local/london
Modern Database #MDBLocal mongodb.com/local/london
Modern Database Document Model Run Anywhere Distributed & Scalable Resilient & Performant #MDBLocal mongodb.com/local/london
Apache Kafka® Minimum Viable Introduction
Streaming Platform #MDBLocal mongodb.com/local/london
Streaming Platform ü distributed ü horizontally scalable ü highly fault-tolerant #MDBLocal mongodb.com/local/london
What is Streaming? “… a type of data processing that is designed with infinite data sets in mind …” – Tyler Akidau #MDBLocal mongodb.com/local/london
“…everything that happens in a company – every customer interaction, every API request, every can be represented as real-time stream that anything else can tap into, process or react to.” database change –
“…Kafka and the whole category of stream processing represents a fundamental paradigm shift in how the digital part of a company is built, how data is used, and how applications are built. This is actually a pretty rare thing…” – Jay Kreps
KStreams App Streams API App Apps n Co Pr App Apps #MDBLocal o c du e P rA I Connect API Connect API Data Sources s e m u P rA Data Sinks I KSQL KSQL App mongodb.com/local/london
Kafka APIs in a Nutshell… • Producer & Consumer API à publish-subscribe scenarios • Connect API à streaming data integration scenarios • Streams API & KSQL à code or SQL-based streaming scenarios #MDBLocal mongodb.com/local/london
Kafka® Connect What’s it about?
Kafka Connect Basics ANY source Connect Connect ANY sink ANY à e.g. file systems, data stores, REST endpoints, … #MDBLocal mongodb.com/local/london
Kafka Connect Basics often about data stores SOURCE DB #MDBLocal Connect Connect SINK DB mongodb.com/local/london
Kafka Connect Basics or more concretely Connect Connect https://hub.confluent.io à many many more #MDBLocal mongodb.com/local/london
Kafka Connect Basics or more concretely Connect Connect https://hub.confluent.io à many many more #MDBLocal mongodb.com/local/london
Kafka® Connect How do connectors operate?
Kafka Source Connectors Source Connector S M T … S M T Converter Serialize 1…N Single Message Transforms for basic in-flight manipulations #MDBLocal mongodb.com/local/london
Kafka Sink Connectors Converter Deserialize S M T … S M T Sink Connector 1…N Single Message Transforms for basic in-flight manipulations #MDBLocal mongodb.com/local/london
Official Connector Announced at #MDBW19
MongoDB Connector for Apache Kafka Map and persist events from Kafka topics directly to MongoDB #MDBLocal Publish data changes from MongoDB into Kafka topics mongodb.com/local/london
MongoDB Connector for Apache Kafka ü developed open-source ü officially supported by MongoDB ü Verified Gold certified by Confluent Available for Download on the Confluent Hub https://www.conflent.io/hub/mongodb/kafka-connect-mongodb #MDBLocal mongodb.com/local/london
Use Cases MongoDB Connector for Apache Kafka
Single Customer View for eCommerce Source Connectors MongoDB Sinks Single Source of Truth #MDBLocal
Data Synchronization between Microservices Service 1 … Service N #MDBLocal MongoDB Sinks
Recommendation Engine for Opinion Mining Change Streams User Recommendation Engine MongoDB Source Surveys & Polls Data Change Streams #MDBLocal
Demo Scenario Let’s see it in action!
Demo Scenario REST data generation device management ! Producer API Change Streams Stream Processor #MDBLocal data serving SSE mongodb.com/local/london
Demo Scenario REST MongoDB Source Connector data generation Producer API device management MongoDB Sink Connector Change Streams Stream Processor #MDBLocal data serving SSE mongodb.com/local/london
THANK YOU!
Streaming Data on the Shoulders of Giants [NETCONOMY] Hans-Peter Grahsl https://www.surveymonkey.com/r/KXC6DB6
Life doesn’t happen in batch mode which is why application engineers and data architects need to closely cooperate to get the best out of streaming platforms like Apache Kafka and operational NoSQL data stores such as MongoDB. This session explores ways and means to integrate both worlds in a streaming fashion.
Without doubt stream processing is a big deal these days and oftentimes we find Apache Kafka as the central nervous system of company-wide data architectures. However, many real-world uses cases simply need an operational data store which is flexible, robust and scalable enough to live up to diverse application-related requirements and challenges. This session discusses different options in order to build solid data integration pipelines between MongoDB and Apache Kafka. The focus lies on configuration-based data in motion scenarios leveraging the Kafka Connect framework in order to lay out streaming ETL pipeline examples without writing a single line of code.
Here’s what was said about this presentation on social media.
for free. You
can too.