Data Engineering and the Death of Visual ETL

A presentation at UKOUG Techfest 2019 in in Brighton, UK by Stewart Bryson

A funny thing happened on the way to modern, data-driven analytics: we ditched the GUI for ETL development. There are drawbacks to moving data using code instead of clicks, especially for organizations heavily invested in visual tools. But writing code has an upside as well.

In this presentation, we’ll explore the data engineering discipline, and how it compares to conventional ETL. We’ll look at classic batch as well as modern streaming use cases, and consider DataOps requirements around testing and delivery where code really shines. And we’ll do it all inside of an interactive notebook.

Resources

The following resources were mentioned during the presentation or are useful additional information.