Is it the end for Apache Airflow?

Tomas Peluritis

The talk will introduce Apache Airflow and its competitors. The main goal of the talk is to showcase how Airflow adapted to the ever-changing data space. Comparison and feature exploration would be oriented in tackling the simplest data extraction and transformation layers in different tools. The plan is to showcase extracting part from Database to S3, move to a database, and add a transformation layer to fill the data warehouse and run data quality checks. The comparison would be on speed, efficiency, integration of different tools and vendors (AWS, dbt Labs, Postgres db) and how it looks in the modern data engineering world (adjustments to more frequent refreshes, dataset awareness, community support), what's crucial, what's missing and how it might go in the future for all of these tools.