All I Need is SQL and a DAG

A presentation at Bored in Quarantine Virtual Series by Stewart Bryson

While enterprise data and analytics software has been abstracting SQL out of our daily lives for decades, distributed data platforms and cloud-native data warehouses have been pushing their chips all-in with SQL, and virtually all Big Data technologies now support it. We all know SQL… so why do enterprise software vendors think it’s such a sin to actually use it?

In this presentation, I’ll describe a modern approach to data integration and ETL using plain SQL and a directed acyclic graph, or DAG. Using the open-source dbt (Data Build Tool) project for demo purposes, we’ll see data and analytics delivered with a modern software discipline including version control and code review, automated testing, and continuous integration and delivery.

Resources

The following resources were mentioned during the presentation or are useful additional information.