A presentation at Dev Innovation Summit 2021 / World Festival by Tim Spann
Continuous SQL with Apache Streaming Timothy Spann Developer Advocate
StreamNative Solution Application Messaging Data Pipelines Real-time Contextual Analytics APP Layer StreamNative Platform Micro Service Payment Notification ETL Dashboard Risk Control Auditing Computing Layer Storage Layer Tiered Storage IaaS Layer streamnative.io
Speaker Bio Developer Advocate DZone Zone Leader and Big Data MVB; @PaasDev https://github.com/tspannhw https://www.datainmotion.dev/ https://github.com/tspannhw/SpeakerProfile https://dev.to/tspannhw https://sessionize.com/tspann/ https://www.slideshare.net/bunkertor streamnative.io
FLaNK and FLiP Stacks ● ● ● Apache Flink Apache NiFi Apache Kafka ● ● ● ● Apache Flink Apache Pulsar StreamNative’s Flink Connector for Pulsar Apache +++ Apache projects are the way for all streaming use cases. streamnative.io
Today’s Data. REST and Websocket JSON I Can Haz Data? {“symbol”:”CLDR”, “uuid”:”10640832-f139-4b82-8780-e3ad37b3d0 ce”, “ts”:1618529574078, “dt”:1612098900000, “datetime”:”2021/01/31 08:15:00”, “open”:”12.24500”, “close”:”12.25500”, “high”:”12.25500”, “volume”:”12353”, “low”:”12.24500”} streamnative.io
End to End Streaming Demo Pipeline Streaming SQL Alerts Enterprise sources Errors Aggregates Analytics Stocks Clickstream Market data Machine logs Social Weather ETL streamnative.io
Multiinges t Multiinges t All Data - Anytime - Anywhere - Multi-Cloud - Multi-Protocol Multi-ingest Merge Priority streamnative.io
Apache Pulsar is Cloud-Native Messaging and Event-Streaming Platform streamnative.io
Apache Pulsar Overview Enable Geo-Replicated Messaging ● ● ● ● ● ● ● ● ● ● ● ● Pub-Sub Geo-Replication Pulsar Functions Horizontal Scalability Multi-tenancy Tiered Persistent Storage Pulsar Connectors REST API CLI Many clients available Four Different Subscription Types Multi-Protocol Support ○ MQTT ○ AMQP ○ JMS ○ Kafka ○ … streamnative.io
What are the Benefits of Pulsar? Multi-Tenancy Unified Messaging Model Scalability Geo-Replication Data Durability streamnative.io
A Unified Messaging Platform Message Queuing Data Streaming streamnative.io
Upcoming - Flink + Pulsar (FLiP) https://flink.apache.org/2019/05/03/pulsar-flink.html https://github.com/streamnative/pulsar-flink https://streamnative.io/en/blog/release/2021-04-20-flink-sqlon-streamnative-cloud streamnative.io
Apache Flink Apache Flink is a distributed stream processing system. It is capable of providing high throughput, near real-time processing of streams from Pulsar. It is ideal for ambitious Stream Processing compared to Pulsar’s model of lightweight Stream Processing. End-to-end exactly-once stream processing https://streamnative.io/en/blog/release/2021-06-14-exactly-once-semantics-with-transactions-in-pulsar streamnative.io
Connect with the Community & Stay Up-To-Date ● Join the Pulsar Slack channel - Apache-Pulsar.slack.com ● Follow @streamnativeio and @apache_pulsar on Twitter ● Subscribe to Monthly Pulsar Newsletter for major news, events, project updates, and resources in the Pulsar community streamnative.io
Deeper Content ● ● ● ● ● ● https://github.com/tspannhw/StreamingSQLExamples https://github.com/streamnative/pulsar-flink https://www.linkedin.com/pulse/2021-schedule-tim-spann/ https://github.com/tspannhw/SpeakerProfile/blob/main/2021/talks/20210729_HailHydrate!FromStream toLake_TimSpann.pdf https://streamnative.io/en/blog/release/2021-04-20-flink-sql-on-streamnative-cloud https://docs.streamnative.io/cloud/stable/compute/flink-sql @PaasDev timothyspann https://www.pulsardeveloper.com/ streamnative.io
Pulsar Summit Europe October 6, 2021 Pulsar Summit Asia November 20-21, 2021 Contact us at partners@pulsar-summit.org to become a sponsor or partner streamnative.io
streamnative.io
18 aug2021 Continuous SQL with Apache Streaming (FLaNK and FLiP) https://emamo.com/event/worldfestival-2021/s/pro-talk-continuous-sql-with-flink-WR115a In this talk, I will walk through how someone can set up and run continuous SQL queries against Pulsar topics utilizing Apache Flink. We will walk through creating Pulsar topics, schemas and publishing data.
We will then cover consuming Pulsar data, joining Pulsar topics and inserting new events into Pulsar topics as they arrive. This basic overview will show hands-on techniques, tips and examples of how to do this using Pulsar tools.
https://github.com/tspannhw/FLiP-IoT https://github.com/tspannhw/SpeakerProfile/tree/main/2021/talks
The following resources were mentioned during the presentation or are useful additional information.
for free. You
can too.