All I Need is SQL and a DAG

While enterprise data and analytics software has been abstracting SQL out of our daily lives for decades, distributed data platforms and cloud-native data warehouses have been pushing their chips all-in with SQL, and virtually all Big Data technologies now support it. We all know SQL… so why do enterprise software vendors think it’s such a sin to actually use it?

In this presentation, I’ll describe a modern approach to data integration and ETL using plain SQL and a directed acyclic graph, or DAG. Using the open-source dbt (Data Build Tool) project for demo purposes, we’ll see data and analytics delivered with a modern software discipline including version control and code review, automated testing, and continuous integration and delivery.