Infrastructure as Python: Pulumi for Cloud Deployments
Speaker
Eric Thanenthiran
I lead Engineering at StellarGlyph, a Data and AI agency. We act as an interim/fractional data team and are passionate about helping clients through building modern data and AI platforms.
With a background in mechanical engineering and have worked across a range of sectors including sustainability, energy, property, construction and architecture. I am an engineer at heart and love building.
Abstract
You use Python for your code why not use it to deploy code to the cloud too? Pulumi, an open source Infrastructure as Code library allows you to configure your cloud infrastructure using Python. In this session we'll deploy a data processing pipeline using Pulumi in GCP and talk through core concepts and fundamentals. Using this example we'll talk through practical best practice ensuring reliable and maintainable infrastructure that scales with your projects.
Description
If you feel using dedicated IAC configuration language like Cloudwatch and Terraform to deploy your projects introduces friction. Pulumi offers a way to do this without leaving the Python ecosystem. Pulumi is an open source Infrastructure as Code library which allows you to write configuration for your infrastructure in many languages, including Python. This gives you all the benefits of a modern language such as conditionals, loops, functions, classes and modules to enable expressive configuration. It also offers the ability to use good software engineering principles more easily and ensure your infrastructure code can scale with complexity.
Getting started is simple and this talk will use an example implementation of a data processing pipeline in GCP. We'll start with an introduction to Pulumi and talk through some core concepts: projects, stacks, config and secrets .We'll cover how to manage state using backends including using local, object storage (S3/Blob/GCS) concurrency and locks. Using common commands we'll look at how to use Pulumi including using previews vs updates, the resource graph and how to check changes and deploy your infrastructure.
Attendees will leave this talk with a solid understanding of how they can get started with Pulumi and how to set it up using best practice.
Outline
- Why Pulumi and Why Python
- Core Concepts
- General setup (projects, state, environment configuration, secrets management)
- Working with Pulummi (commands, resource components, change management)
- Data processing service project
- Project Structure
- Best Practice and Pitfalls