Modern Python monorepo for Apache Airflow
Speaker
Jarek Potiuk
Independent Open-Source Contributor and Advisor, Committer and PMC member of Apache Airflow, Member of the Apache Software Foundation, Security Committee Member of the Apache Software Foundation. Organizer of community-focused events, speaker.
Jarek is an Engineer with a broad experience in many subjects - Open-Source, Cloud, Mobile, Robotics, AI, Backend, Developer Experience, Security, but he also had a lot of non-engineering experience - building a Software House from scratch, being CTO, organizing big, international community events, technical sales support, pr and marketing advisory but also looking at legal aspects of security, licensing, branding and building open-source communities are all under his belt.
With the experience in very small and very big companies and everything in-between, Jarek found his place in the Open Source world, where his internal individual-contributor drive can be used to the uttermost of the potential.
Abstract
Explore how Apache Airflow modernized its massive monorepo by transitioning from complex custom scripts to official Python packaging standards and tools like uv. This session breaks down the management of 120+ distributions, the implementation of modular pre-commit hooks, and a novel approach to "static" shared libraries within a single repository. Join us to see a real-world blueprint for large-scale modularity as we advocate for the formalization of a Python workspace standard.
Description
Historically, managing large-scale projects within a monorepo was a significant challenge, particularly when multiple distributions shared complex internal dependencies. For over five years, Apache Airflow—one of the largest Python projects in existence by contributor volume—relied on a complex, custom-built system to manage and release nearly 100 distributions from a single repository.
However, the landscape has shifted. Recent leaps in Python packaging standards and tooling have revolutionized the ecosystem. By adopting modern PEPs and cutting-edge tools like uv and prek, Airflow has transitioned to a highly manageable monorepo supporting 120+ distributions. This shift has not only improved the developer experience but also paved the way for further modularization—including a novel approach to sharing common code that mirrors the behavior of static system libraries in Unix-like systems.
We are eager to share our journey and contribute to future PEP discussions, offering Airflow’s success as a blueprint for modern, large-scale monorepo management in Python.