Data Version Control Done Right with Python and Unity

Schedule

Room: Room 203

April 5

15:00–15:25

Abstract

Python is a leading language of choice for the Databricks and ML ecosystem, alongside a delta tables stack leveraging Unity catalog to manage petabytes of structured data. To build and experiment with ML data and models, version control has become the backbone of modern machine learning (ML) projects, bringing critical aspects of reproducibility and experimentation to teams who are able to experiment in isolation, while still collaborating on projects.

Prerequisites

Python, Data Engineering Concepts, ML Concepts

Description

One critical area that version control ensures is greater data compliance with ACID principles when it comes to transactions.

In this talk we'll demo reproducibility and efficient experimentation management in Python using delta tables managed in Unity. We'll walk through a Python example of how to simulate transactions in lakeFS with a dedicated Python SDK. Branching mechanisms make it possible to manage changes in the data lake, while simulating transactional concepts in data lakes in order to track changes over time and, as well as provide the guardrails of reverting to a previous state when needed for an added level of consistency and traceability.