Write-Audit-Publish Pattern in Modern Data Pipelines
The talk
Schedule
Room: Room 203
April 5
11:30–11:55
Abstract
Data is new oil, and one of the ways is leakage and poisoning the surrounding environment. What happens if you pollute one of the datasets used in some decision makers facing dashboards? In this talk, I will explain the reemergence of the Write-Audit-Publish pattern and how you can achieve it using Apache Iceberg and Apache Spark.
Prerequisites
General knowledge, Data Processing
Description
One of the old patterns that is being adopted again is Write-Audit-Publish. Most Data Practitioners are testing the data they create after it’s done, potentially ruining other data assets downstream. Using WAP allows us to ensure that no such thing happens. In this talk, I will present how evolving new tech (in this case, Apache Iceberg table format) allows us to use Apache Spark to use this pattern. The presentation will cover topics like:
- What is Write-Audit-Publish
- What is Apache Spark and How it works with Apache Iceberg
- What is the Branching feature in Apache Iceberg, and how does it connect to WAP
- Example of its use together with Apache Spark
