A presentation at ODSC Europe by Marta Paes
© 2020 Ververica
2
open source distributed stateful stream processing Flink Runtime 3
open source distributed stateful stream processing High Performance Fault Tolerance Stateful Processing Flink Runtime 4 Flexible APIs
Streaming Analytics & ML Stateful Stream Processing Flink Runtime 5 Event-Driven Applications
streams state Streaming Analytics & ML Stateful Stream Processing Flink Runtime 6 time Event-Driven Applications
streams state ● ● ● 7 performance reliability time
Streaming Analytics & ML Stateful Stream Processing Flink Runtime 8 Event-Driven Applications
● ● ● 9 developer speed autonomy
10
11
Mature fast 12 intuitive
1995 2008 Mature 2003 2015 2001 13 fast intuitive
1995 2008 Mature 2003 2015 fast intuitive 2001 restricted fits in memory single core 14
15
16
17
18
Expose 19
Distribute 20 scale
Double 11 / Singles Day 21
● ● ● 22
● ● ● Execution Streaming Batch UDF Support 23 Python UDF Pandas UDF +UDAF (WIP) +UDAF (WIP)
● ● ● Execution Streaming Native Connectors Batch FileSystems Apache Kafka ML Library (WIP) FLIP-39 Notebooks UDF Support Kinesis Python UDF Pandas UDF +UDAF (WIP) +UDAF (WIP) HBase JDBC Elasticsearch
● ● ● Execution Streaming Native Connectors Batch FileSystems Apache Kafka ML Library (WIP) FLIP-39 Notebooks UDF Support Kinesis Python UDF Pandas UDF +UDAF (WIP) +UDAF (WIP) HBase JDBC Elasticsearch
26
… Advantages ● ● ● ● ● 27
DEMO 28
DEMO 29
DEMO 30
DEMO 31
DEMO 32
DEMO 33
34
© 2020 Ververica
Stream processing has fundamentally changed the way we build and think about data pipelines — but the technologies that unlock the value of this powerful paradigm haven’t always been friendly to non-Java/Scala developers. Apache Flink has recently introduced PyFlink, allowing developers to tap into streaming data in real-time with the flexibility of Python and its wide ecosystem for data analytics and Machine Learning. In this talk, we will explore the basics of PyFlink and showcase how developers can make use of a simple tool like interactive notebooks to harness the full power of an advanced stream processor like Apache Flink.
The following resources were mentioned during the presentation or are useful additional information.
A hands-on example to get you started from scratch.
Here’s what was said about this presentation on social media.
for free. You
can too.