Technical Debt: When to Pay It Down vs. When to Just Live With It
Speaker
Tomas Peluritis
Tomas leads data at Mediatech and runs Uncle Data, a newsletter and podcast for data engineers who prefer practical advice over hype. By day, he manages pipelines processing half a billion events; by night, he writes about what he learned (often the hard way). When not wrangling DAGs or mentoring his team, he's probably optimising a Magic: The Gathering deck.
Abstract
Every team has code that makes them cringe. Legacy pipelines nobody dares touch, "temporary" tables that became the source of truth, and SQL queries that outlived three team leads. We call it technical debt, and the instinct is always the same: fix it.
But should we? After migrating from Redshift to Snowflake, learning PIG just to rewrite a pipeline deprecated months later, and surviving Airflow OOM kills, I have opinions. And maybe a framework slightly better than "it depends."
Description
Every engineering team has code in production that makes them cringe. Legacy pipelines nobody dares touch, temporary tables that somehow became the "single source of truth," and SQL queries over a thousand lines long that have outlived three team leads. We call it technical debt, and the instinct is always the same: we should fix it.
But should we? And if so, when? Because I've been on both sides. The "let's rewrite everything properly" crusade and the "if it works, don't touch it" survival mode. Spoiler: neither is always right. After years of leading data teams processing hundreds of millions of daily events, I've learned that sometimes the bravest engineering decision is doing nothing. And sometimes the laziest-looking decision is actually the smartest refactor you'll ever make.
In this talk I'll share real stories. Migrating from self-managed Airflow on EC2 to MWAA. Moving from Redshift to Snowflake. Learning PIG just to rewrite a pipeline that got deprecated five months later. Each one came with tradeoffs nobody warned me about.
We'll dig into Martin Fowler's Debt Quadrant, a Pain vs. Cost matrix for prioritising what actually matters, how to translate technical debt into business language that gets stakeholder buy-in, and what changes when AI tools make refactoring cheaper but introduce entirely new categories of debt.
You'll walk away with a decision framework you can apply on Monday, a template for cataloguing your team's debt, and the confidence to tell your manager, or yourself, that some debt is perfectly fine to live with.
No silver bullets. Maybe a framework that's slightly better than "it depends." No promises on that last part.