Is it the end for Apache Airflow?
Room: Saphire B - PyData
Date: 2023-05-19
Time: 12:00 - 12:25
Abstract
The talk will introduce Apache Airflow and its competitors. The main goal of the talk is to showcase how Airflow adapted to the ever-changing data space. Comparison and feature exploration would be oriented in tackling the simplest data extraction and transformation layers in different tools. The plan is to showcase extracting part from Database to S3, move to a database, and add a transformation layer to fill the data warehouse and run data quality checks. The comparison would be on speed, efficiency, integration of different tools and vendors (AWS, dbt Labs, Postgres db) and how it looks in the modern data engineering world (adjustments to more frequent refreshes, dataset awareness, community support), what's crucial, what's missing and how it might go in the future for all of these tools.
Tomas Peluritis
BI developer turned to Data Engineer with experience in data analysis and data science. Have experience with MS solutions to a lot of open-source tools work-wise. Free time - exploring different options to improve the company's infrastructure/toolset or learning new tech by writing about it.
Currently more focusing to the management track, though can't shake the need to get my hands dirty.
Podcasting and writing in English (on medium) and Podcasting about data (Lithuanian)
