A presentation at Flink Forward Global 2020 by Marta Paes
Building an E2E Analytics Pipeline with PyFlink Marta Paes (@morsapaes) Developer Advocate © 2020 Ververica
This Talk Motivation & Evolution of PyFlink 2 @morsapaes Demo Looking ahead
Why Flink + Python? Java Scala 3 @morsapaes
Why Flink + Python? Java Python Scala Expose the functionality of Flink beyond the JVM 4 @morsapaes
Why Flink + Python? Mature and intuitive analytics stack… 5 @morsapaes
Why Flink + Python? Mature and intuitive analytics stack… 2007 1995 2001 2008 …that doesn’t really scale. 6 @morsapaes
Why Flink + Python? 2007 1995 2001 Distribute and scale the functionality of Python through Flink 2008 7 @morsapaes
PyFlink Over Time PyFlink (Table API) Beta release Flink 1.9 Aug’19 8 @morsapaes
PyFlink Over Time PyFlink (Table API) Beta release Flink 1.10 Feb’20 Flink 1.9 Aug’19 Scalar UDFs apache-flink available on PyPi 9 @morsapaes
PyFlink Over Time Python UDFs in SQL DDL & SQL Client Tabular UDFs UDF metrics Pandas UDFs & Cython UDF Optimization PyFlink (Table API) Beta release Table conversion fromPandas/toPandas Flink 1.10 Flink 1.12 Feb’20 Nov/Dec’20 Flink 1.9 Flink 1.11 Aug’19 Jul’20 Scalar UDFs apache-flink available on PyPi 10 @morsapaes
Can we use PyFlink to identify the most frequent topics in the User Mailing List? 11 @morsapaes
The Demo Environment CDC Source Connector PyFlink * Postgres Submit job UDF Visualization JobManager Assign & monitor tasks
DEMO Step 1. Create the source table. 13 @morsapaes
DEMO Step 2. Write and register a UDF to clean and classify the messages. 14 @morsapaes
DEMO Step 3. Build your query and create a sink table to where you’ll be inserting the results! 15 @morsapaes
DEMO Step 4. Submit the job (and dependencies) to the cluster docker-compose exec jobmanager ./bin/flink run -py /opt/pyflink-ff2020/pipeline.py \ —pyArchives /opt/pyflink-ff2020/lda_model.zip#model \ —pyFiles /opt/pyflink-ff2020/tokenizer.py -d 16 @morsapaes
DEMO Step 5. Visualize in Superset! (9, ‘0.521*”flink” + 0.065*”issue” + 0.049*”source” + 0.041*”yarn” + 0.033*”build” + 0.028*”latency” + 0.027*”access” + 0.023*”progress” + 0.016*”run” + 0.015*”org”’) 17 @morsapaes
And soon! Python UDFs in SQL DDL & SQL Client Tabular UDFs UDF metrics Pandas UDFs & Cython UDF Optimization PyFlink (Table API) beta release Table conversion fromPandas/toPandas Flink 1.10 Flink 1.12 Feb’20 Nov/Dec’20 Flink 1.9 Flink 1.11 Aug’19 Jul’20 Scalar UDFs Python DataStream API (Stateless) [FLIP-130] apache-flink available on PyPi PyFlink on (native) Kubernetes [FLINK-17480] UDAFs & Pandas UDAFs [FLIP-139, FLIP-137] Python Table API DSL [FLINK-19091] 18 @morsapaes
PyFlink in a Nutshell ● Unified APIs for batch and streaming ● Native SQL integration + Pandas integration ● Support for a large set of operations (incl. complex joins, windowing, pattern matching/CEP) Execution Streaming Native Connectors Formats Batch FileSystems Apache Kafka ML Library (WIP) FLIP-39 Notebooks UDF Support Kinesis Python UD(T)F Pandas UDF +UDAF (WIP) +UDAF (WIP) HBase JDBC Elasticsearch Apache Zeppelin
Thank you, Flink Forward! Follow me on Twitter: @morsapaes Learn more about Flink: https://flink.apache.org/ © 2020 Ververica
Stream processing has fundamentally changed the way we build and think about data pipelines — but the technologies that unlock its value haven’t always been friendly to non-Java/Scala developers.
Flink has recently introduced PyFlink, allowing developers to tap into streaming data in real-time with the flexibility of Python and its wide ecosystem for data analytics and Machine Learning. In this talk, we’ll explore the basics of PyFlink and showcase how developers can make use of familiar tools like interactive notebooks to unleash the full power of an advanced stream processor like Flink.
Here’s what was said about this presentation on social media.
for free. You
can too.