Beyond dbt: Modern SQL Transformation and Lineage with sqlglot and sqlmesh

Abstract

Hear more about the evolving landscape of SQL transformation tools and data lineage challenges. Explore how sqlglot enables powerful SQL parsing and transformation capabilities, and see practical demonstrations of sqlmesh as a modern alternative to dbt. Learn about open-source approaches to data lineage tracking and discover how these tools are shaping the future of data engineering workflows.

Prerequisites

SQL, Python

Description

This talk explores the intersection of SQL transformation frameworks and data lineage tracking, focusing on open-source solutions that are changing how we handle data transformations at scale. We'll begin by examining common pain points in data lineage tracking, particularly when dealing with complex SQL transformations across different dialects and platforms.

The first part will deep dive into sqlglot's architecture and demonstrate how it serves as a crucial building block for modern data tools by enabling dialect-agnostic SQL parsing, analysis, and transformation. We'll explore real-world use cases where sqlglot's capabilities unlock new possibilities for data lineage tracking and SQL optimization.

Next, we'll contrast sqlmesh with dbt, highlighting key architectural differences and their implications for data engineering workflows. Through live demonstrations, we'll showcase sqlmesh's unique features including time-travel capabilities, automated dependency management, and built-in data lineage tracking. We'll also address how these tools approach column-level lineage tracking, comparing open-source alternatives to proprietary solutions like dbt Cloud.

The session will conclude with practical guidelines for implementing these tools in your data stack and a discussion of future trends in SQL transformation and lineage tracking. Whether you're a data engineer looking to optimize your workflow or an architect evaluating data transformation frameworks, you'll leave with actionable insights about modern SQL tooling.