Airflow Lessons They Don't Put in the Docs
Speaker
Tomas Peluritis
Tomas leads data at Mediatech and runs Uncle Data, a newsletter and podcast for data engineers who prefer practical advice over hype. By day, he manages pipelines processing half a billion events; by night, he writes about what he learned (often the hard way). When not wrangling DAGs or mentoring his team, he's probably optimising a Magic: The Gathering deck.
Abstract
Airflow basics are well documented. Production Airflow is not. This talk covers the patterns, costs, and migration pitfalls that only show up after you've deployed: dynamic DAGs that scale, sensors that don't waste resources, CloudWatch bills that surprise you, and MWAA version upgrades that break in ways the changelog didn't mention. Practical lessons for teams running Airflow beyond the tutorial stage.
Description
You've deployed Airflow, your DAGs are running, and the basics are under control. But then reality hits: dynamic DAGs that bring your scheduler to its knees, sensors that quietly burn through resources, and a CloudWatch bill that rivals your compute costs. This talk goes beyond the getting-started guides. We'll cover dynamic DAG patterns that actually scale, when custom operators are worth the investment, and sensor anti-patterns that plague production systems. For teams running on AWS MWAA, we'll dive into the undocumented pain points: CloudWatch metrics that silently drain your budget, version upgrade migrations that break in unexpected ways, and autoscaling behaviour the docs don't fully explain. You'll leave with practical patterns, cost-saving strategies, and a checklist for running Airflow in production without the surprises.