Data Engineering and the Death of Visual ETL

A presentation at Atlanta Snowflake, Big Data and Spark Joint Meetup in in Atlanta, GA, USA by Stewart Bryson

A funny thing happened on the way to modern, data-driven analytics: we ditched the GUI for ETL development. There are drawbacks to moving data using code instead of clicks, especially for organizations heavily invested in visual tools. But writing code has an upside as well.

In this presentation, we’ll explore the data engineering discipline, and how it compares to conventional ETL. Using Databricks and Snowflake together, we’ll look at classic batch and modern streaming use cases, and consider DataOps requirements around testing and delivery where code really shines. And we’ll do it all inside of an interactive notebook.

Resources

The following resources were mentioned during the presentation or are useful additional information.