Company presentations

Lightning talks

Company presentations 2

Travis Oliphant

Leader talks

Samuel Colvin

Uncle Data

Uncle Data session 1

Justinas Kuizinas

Uncle Data session 2

Marc Garcia

Ritchie

Uncle Data session 3

Would you like to contribute to open source projects and you don't know where to start? In this talk we will show you how. It is easier than it sounds when you know the basics on how GitHub is used for open source projects.

Make your first open source contribution

Every business is unique, as are its data and problems. In order to make the most of our valuable data, we need to incorporate these intricacies when building a solution. Bayesian modeling provides a powerful framework for building a so-called digital twin that maps our real-world problem and domain-expertise into a statistical model that can then be fit to data. The benefits are that our solutions are transparent and interpretable by stakeholders, and come with uncertainty measures. 
 
In this talk I will give a few examples of real-world business problems and how Bayesian modeling can solve them. In particular, some common patterns observed in business data sets are time-series, hierarchical or nested structure, and spatial data. 

In this talk I will give a few examples of real-world business problems and how Bayesian modeling can solve them. In particular, some common patterns observed in business data sets are time-series, hierarchical or nested structure, and spatial data.

Dr. Thomas Wiecki

Bayes in Business: Transparent and Interpretable Solutions

[Pydantic](https://docs.pydantic.dev/) is a data validation library for Python that has seen massive adoption over the last few years - it's used by major datascience and ML libraries like [Spacy](https://spacy.io/usage/v3#features-types), [Huggingface](https://github.com/huggingface/transformers) and [jinja-ai](https://github.com/jina-ai/jina) - overall Pydantic is downloaded over 55m times a month!

In this talk Samuel Colvin, the creator of Pydantic will cover two subjects which have seen massive interest in recent years:

* How Pydantic can be used to prepare data for machine learning thereby saving time and avoiding errors
* The emergence of Rust as the go-to language for high performance python libraries - how this might go in the future, and the benefits and drawbacks of the trend

In this talk I'll give a brief introduction to Pydantic, what it can do and how it differs from other similar libraries.

I'll then go on to walk through an example of how Pydantic can be used to prepared data to train a machine learning model, including some advantages of Pydantic over dataclasses or regular dictionaries.

Finally I'll give a high level introduction to how Rust is being used to build python extensions, and why that's (mostly) a great thing for the community and the planet. The two main case studies will be the recent [re-write of Pydantic in Rust for V2](https://docs.pydantic.dev/blog/pydantic-v2/), and [Polars](https://www.pola.rs/).

Garbage in -> Pydantic -> you're golden!

In this talk we will examine how Python programmers can be creative in the digital age. Whether it's in web development, data science or machine learning creativity is a skill that sets a good developer apart from a great one. Python as a language has been around for 32 years and it may seem like the old saying 'there is nothing new under the sun' is true since so much has been built with it. For this talk we'll look at why building off of the work of other people is essential for creativity and how open source is the thread that ties it all together.

Marlene Mhangami

Python and Creativity (An Explorers Guide)

For 28 years, the "SciPyData" community has been building foundational software that has influenced science, and engineering and jumpstarted the recent explosion of interest in AI and Machine Learning.   in this talk I will briefly review key milestones in the creation of SciPy and PyData communities and discuss key future opportunities and advances in the foundational software behind AI and ML that could effect the next 5 to 10 years.

Building sustainable software for AI and ML with lessons from the SciPyData community

The talk will introduce Apache Airflow and its competitors. The main goal of the talk is to showcase how Airflow adapted to the ever-changing data space. Comparison and feature exploration would be oriented in tackling the simplest data extraction and transformation layers in different tools. The plan is to showcase extracting part from Database to S3, move to a database, and add a transformation layer to fill the data warehouse and run data quality checks. The comparison would be on speed, efficiency, integration of different tools and vendors (AWS, dbt Labs, Postgres db) and how it looks in the modern data engineering world (adjustments to more frequent refreshes, dataset awareness, community support), what's crucial, what's missing and how it might go in the future for all of these tools.

Tomas

Is it the end for Apache Airflow?

More than 5 million people play Nordcurrent mobile games every month. The specificity of free-to-play games is that less than 10% of players make purchases. It is essential to retain paying players and keep them engaged as long as possible. To do that, we built a purchase prediction model.

We store data and make the most of feature engineering in Clickhouse. Apache Airflow orchestrates pipelines. Usually, we use CatBoost for Machine Learning. Pydantic and ClearML, on top of AWS S3, manage model files, training metrics, and configs. The quality in production is evaluated using dashboards in Apache Superset.

The architecture allows us to build fully reproducible ML pipelines. The learning process can be horizontally scaled to select the optimal hyperparameters. At the inference stage, you do not need to worry that the model was trained in some Jupiter Notebook, and it is unclear what to do if it suddenly breaks in a month.

Dima Savostyanov

How we predict purchases in mobile games

Most data transformations are written twice. In the field of feature engineering for Machine Learning, data scientists regularly have to build, manage, and iterate on batch jobs, then translate those jobs to a service setting to load data and make fresh predictions. At best, this process is an engineering headache. At worst, this can result in difficult-to-detect deltas between training and inference, complex code, and highly bespoke infrastructure. In this talk we discuss Hamilton, a lightweight open-source framework in python that enables data practitioners to cleanly and portably define dataflows. Hamilton places no restrictions on the nature of transformations, allowing data scientists to use their favorite python libraries. With Hamilton, you can run the same code in your airflow DAG for training as you would in your fastAPI service for inference, and get the same result.

In this talk, we present Hamilton, and talk about how it can enable data scientists to build highly portable dataflows that can run in a variety of different contexts. At a high level, we will discuss:
The paradigm Hamilton introduces, and how it is simplifies the process of building and maintaining feature engineering pipelines
How Hamilton can be used to help scale batch data preparation for training and inference
How the same hamilton code can be used in a web-service to prepare data and run live inference, with minimal changes

We will go over working code examples, making sure to connect with tooling people are familiar with (E.G. airflow, fastapi, metaflow, django).

Elijah ben Izzy

Portable Feature Engineering with Hamilton: Write Once, Run Everywhere

In this talk, we will be exploring what memory profiling is, and how it can help with data science work. We will start the talk with a basic explanation of how Python arrange memories for various objects. This lays the foundation explanation of why we need a special tool to memory profile Python programs.

Then we will be going through a data science use case where we memory profiles some part of the process with the Memray Jupyter plug-in. This would be a use case that a data science practitioner or learner would be familiar with and they can see how memory profiling could be useful. 

We will then explain how to interpret the frame diagram in Memray, a commonly used diagram in memory profiling to understand how much memory a process and its sub-process uses. This is something that for a new user, it could be hard to understand and not know what to look into. From this example, audiences can see what they can learn about from the frame diagram.

## Goal

This talk is for data scientists, learners or anyone who is interested in memory profiling their Python program. Although the talk will be using a data science use case as an example, the explanation and the tool can be expanded to be used in any Python program. However, for data science practitioners and learners who have been using Python to process data, this may be a step forward for them to improve their data workflow and prevent memory leaks from their programs.

## Outline

- Introduction (5 mins)
- Why we need a special tool for memory profiling (5 mins)
- How to use Memray in Jupyter notebook (5 mins)
- Demonstration for using Memray in data science work (5 mins)
- How to interpret a frame diagram (5 mins)
- Conclusion (5 mins)

Cheuk Ting Ho

Driving down the Memray lane - Profiling your data science work

Building a strong data science portfolio can be a daunting task, especially for those just starting out in the field. 
In this talk, we will explore the essential elements of a successful data science portfolio, and provide actionable advice for building a portfolio that will make recruiters take notice.

First, we will discuss the importance of selecting the right projects for your portfolio. We'll share tips for identifying projects that demonstrate a range of skills and showcase your expertise in a specific area. 

Next, we'll talk about how to present your work in a clear and compelling way, including how to structure your portfolio and which tools and platforms to use.

We will also discuss how to incorporate feedback from peers and mentors, as well as how to solicit feedback from potential employers. In addition, we'll cover best practices for maintaining and updating your portfolio, and how to use your portfolio to continue learning and growing in the field of data science.

Karolina Griciunė

How to Build a Data Science Portfolio That Will Make Recruiters Swipe Right

In this talk, we will explore the rapidly evolving field of MLOps. I will delve into best practices and tools that are essential for building and deploying machine learning models. I will cover topics such as data management, hyperparameter tuning, model training and model deployment, also share the latest techniques and tools for streamlining these processes and discuss best practices for monitoring and maintaining machine learning models in production. Whether you are a seasoned machine learning practitioner or just starting out, this talk will provide insights and practical tips for building robust, scalable, and maintainable machine learning systems. Join me as we dive into the world of MLOps and explore the tools and techniques that every machine learning engineer should know.

Aurimas Griciunas

MLOps Fundamentals or What Every Machine Learning Engineer Should Know

MLOps tools today are dime a dozen, but do you really need everything to build your machine learning pipelines? If you are just getting started you do not need an army of tools to set up your ML pipelines. In this talk, I will introduce you to the general concept of MLOps, why it is becoming more important these days and then focus on a super interesting MLOps framework in Python called ZenML. ZenML helps you structure your code and pipelines systematically right from the word go, ensuring that you are always building pipelines that can be easily deployed in production. ZenML has a lot of custom components that can be used in different ways. I will take you through the many concepts (steps, pipelines, stacks, integrations) used by ZenML and how you could use them to build your production ready Machine Learning pipelines.

MLOps tools today are dime a dozen, but do you really need everything to build your machine learning pipelines? In this talk, I will introduce you to the general concept of MLOps and then focus on a super interesting MLOps framework in Python called ZenML. ZenML helps you structure your code and pipelines systematically right from the word go, ensuring that you are always building pipelines that can be easily deployed in production. I will take you through the many concepts (steps, pipelines, stacks,integrations) used by ZenML and how you could use them to build your production ready Machine Learning pipelines.

Though the structure is tentative, I intend to follow the following order:
1. Introduction to MLOps
2. MLOps Lifecycle
3. Introduction to ZenML concepts
4. ZenML Architecture
5. ZenML pipeline creation
6. Switching stacks to deploy a machine learning pipeline to production
7. Question and Answers

Imaad Mohamed Khan

Production ready Machine Learning pipelines using ZenML for MLOps management

Are you struggling with big data in your business? Join us to discover how PySpark can help you solve your problems efficiently and effectively. In this workshop, we will revisit the key concepts of PySpark, including parallel processing and lazy evaluation. We will explore DataFrames as a convenient layer of so called RDDs and work with an optimizer to get the most out of our transformations. 

We'll also take a look the Spark UI, which allows us to monitor and optimize our processes. To put our knowledge into practice, we'll simulate a business problem and walk through the entire process of data preparation (preprocessing), training a model with MLLib, and performing inference on preprocessed test data. We'll also add a business logic layer to our solution for further customization (postprocessing). 

Optional content includes lessons learned from large-scale production systems based on PySpark. We'll share insights on how to optimize performance and scale your solution to handle big data with ease.

Are you looking for a powerful tool to tackle your big data problems? PySpark may be just what you need. Join us for a comprehensive workshop on PySpark, where we'll cover everything from the basics of parallel processing and lazy evaluation to deploying production-level solutions on a large scale. 

In this workshop, we'll start by visiting the key concepts of PySpark and exploring how it can help us solve business problems with potentially very large amounts of data. We'll then dive into working with DataFrames as a convenient layer of RDDs and utilizing an optimizer to get the most out of our data. 

To help you put your new knowledge into practice, we'll simulate a real-world business problem and walk you through the entire process of data preparation, model training with MLLib, and performing inference on preprocessed test data. We'll also add a business logic layer to our solution for further customization. 

Throughout the conference, we'll utilize the Spark UI to monitor and optimize our processes. We'll provide you with code that can be easily adapted to run on various platforms, including a cluster in AWS Glue and localhost. 

In addition, we'll cover optional content on lessons learned from large-scale production systems based on PySpark. We'll share insights on how to optimize performance and scale your solution to handle big data with ease. 

Whether you're just starting out with PySpark or looking to take your skills to the next level, this workshop is designed for you. Join us to discover how to harness the power of PySpark for big data and take your business to the next level.

Carsten Frommhold

Unlocking the Power of PySpark: A Comprehensive Workshop

Are you curious about how recommendation engines work? In this workshop, we'll dive deep into the world of TensorFlow Recommender Systems, exploring the fundamental concepts, techniques, and tools needed to build effective recommendation engines. We'll start with an overview of the different types of recommender systems, including collaborative filtering, content-based filtering, and hybrid models. We'll also explore evaluation metrics and learn how to measure the effectiveness of these models.
The workshop then shifts to hands-on exercises that allow you to build your own recommendation engine using TensorFlow. You'll learn how to prepare data, train the model, and make recommendations. Through guided examples, you'll gain a practical understanding of the end-to-end process of building a recommendation engine.

In today's digital world, personalized recommendations have become an essential part of user engagement and retention. With the abundance of data available, building effective recommendation systems can seem daunting. In this workshop, we will take a deep dive into the world of TensorFlow Recommender Systems and explore the techniques and tools necessary to build a high-performing recommendation engine.

We will start by understanding the fundamentals of recommender systems and how TensorFlow can be used to build them. We will then explore the different types of recommenders, including content-based, collaborative filtering, and hybrid models. We will also delve into the various evaluation metrics used to measure the effectiveness of recommender systems.

In the second half of the workshop, we will get hands-on experience with TensorFlow Recommender Systems. We will work through a real-world use case and learn how to prepare the data, build and train the model, and make recommendations. 

By the end of the workshop, attendees will have a solid understanding of the key concepts and tools required to build powerful recommender systems using TensorFlow. They will have the opportunity to apply what they have learned and create their own personalized recommendations. Whether you're a data scientist, engineer, or product manager, this workshop is the perfect opportunity to dive deeper into the world of TensorFlow Recommender Systems and take your skills to the next level.

Ashmi Banerjee

The Ultimate Matchmaker: Building Recommender Systems with TensorFlow

DataFrame abstractions are one of the favorite data structures of many data scientists, data-engineers and programmers in general. They offer flexibility and intuitive reasoning on top of query processing.

However, the implementation of DataFrame abstractions have been lacking. On the single node they have been ignoring most research available in RDBMS research. Different from RDBMS, the most known python implementations don't control their own query engines, and are therefore always compromising control, performance and memory usage.

Polars is a DataFrame library that brings a very fast OLAP query engine to the DataFrame abstraction. 

This talk we look at what polars has achieved since it's inception and what the future will hold in store.

<some extra characters because they were needed to fill the cell>

Polars: done the fast, now the scale

From recommendation systems to LLM-based applications, vector search is a critical component of the modern AI workflow. Existing vector solutions are complicated to use, hard to maintain, and cost too much. LanceDB is a free open-source vector store that can perform low latency vector search on billion-scale vector datasets on a single node. LanceDB is powered by Lance format, a modern columnar data format for machine learning and data science. Compatible with pandas/polars/duckdb, Lance format supports vector index, predicate pushdown, and random access performance 2000x faster than parquet.

This talk will:
1. Introduce LanceDB and show some example workflows
2. Outline Lance format design and what makes it so fast
3. Review the Lance roadmap and ecosystem integrations

You can find Lance here: https://github.com/eto-ai/lance

Chang She

Serverless billion-scale vector search for AI applications

Building Streamlit apps that enable business users to explore data on their own is an excellent way to support data-driven decision-making in any organization. Pair this up with the animated transitions between the charts provided by the new, open-source library, ipyvizzu, and you have a self-service, interactive report or dashboard that makes it much easier for non-experts to make sense of complex data sets.

It's great when you can share the results of your analysis not only as a presentation but as something that non-data scientists can explore on their own, looking for insights and applying their business expertise to understand the significance of what they find.

With its accessibility for both creators and viewers, Streamlit offers a brilliant platform for data scientists to build and deploy data apps. Now, with the integration of [ipyvizzu](https://ipyvizzu.com) - a new, open-source data visualization tool focusing on animation and storytelling - you can quickly create and publish interactive, animated reports and dashboards on top of static or dynamic data sets and your models.

In this talk, one of the creators of ipyvizzu shows how their technology works within Streamlit and the advantages of using animation in self-service data exploration to help business stakeholders feel smarter and do a better job.

Peter Vidos

Let them explore! Building interactive, animated reports in Streamlit with ipyvizzu & a few lines of Python

Writing maintainable data science code is a big topic, and different people have different opinions on the best ways to do it. Wouldn't it be nice if there was an opinionated framework to set some structure and help data scientists be more effective and ship their analysis and models to production faster?

In this workshop we present Kedro, an opinionated Python framework for creating reproducible, maintainable and modular data science code. We will also show how you can combine it with Polars, a new dataframe library backed by Arrow and Rust, for lightning fast data manipulation and exploratory data analysis.

In this workshop we present Kedro, an opinionated Python framework for creating reproducible, maintainable and modular data science code. We will also show how you can combine it with Polars, a new dataframe library backed by Arrow and Rust, for lightning fast data manipulation and exploratory data analysis.

Kedro is an open source (Apache 2.0) Python framework for maintainable data science that provides a series of project templates, a declarative data catalog, functionality to create function-based data pipelines, and a powerful visualization tool. It has a rich ecosystem of plugins and extensions and a thriving community.

Traditionally, Kedro has encouraged the use of pandas for data I/O and manipulation. In recent times, Polars has become increasingly popular thanks to its expressive API, its lazy evaluation system, its out of core capabilities, and its impressive performance.

The workshop will be hands on, and the outline is as follows:

1. The problem of maintainability in data science code
2. What is Kedro?
3. Quick data I/O with Polars
4. Introducing the Kedro catalog and the Jupyter integration
5. Creating pipelines in Kedro
6. More exploratory data analysis with Polars
7. Plots in Kedro Viz

Juan Luis Cano Rodríguez

Analyze your data at the speed of light with Polars and Kedro

pandas 2.0 has recently been released, and one of the key features is a greater support of the Apache Arrow in-memory format. While the change is somehow internal, it opens a wide range of possibilities. In this talk we will have a quick overview of pandas and Apache Arrow, what is new in pandas 2.0, how users will be able to benefit from using pandas with Apache Arrow and what to expect from future pandas releases.

pandas 2.0 and the Arrow revolution

AI technologies brough the ability to construct the data in new light. Search engines learn to search not by keywords but by semantic information, the same goes for image or audio search. In this talk we dive into idea and realization how to construct Image/Text/Audio search from scratch, using existing AI pipelines.

We dive into usage of Pytorch library and existing pre-trained models, to construct Image/Text/Audio search from scratch, using existing AI pipelines. The investigation of paralelization of computations using numba (cuda) library will expand the speeup cases.

Linas Petkevičius

Similarity search in practice or how AI can help in everyday’s life

In the current age of AI, the ability to rapidly develop and deploy applications has become crucial for staying competitive. As the demand for AI-powered solutions continues to grow, it's more important than ever to be able to bring new ideas to market quickly and efficiently. The H2O Wave framework is a powerful tool that enables DS/ML people with the required business knowledge to do just that, without the unnecessary overhead of having a software engineering team in the middle.

This talk will introduce H2O Wave, a Python framework that allows developers to build web applications with minimal web development knowledge. With its high-quality UI widgets, built-in authentication, and developer tooling such as IDE extensions, H2O Wave simplifies the app development process and helps teams bring their AI-powered applications to market faster. Attendees will learn how the framework is already being used by Kaggle Grandmasters to build AI applications and how it can help their own development efforts.

Martin Turóci

H2O Wave - Build web apps with nothing but Python

The word ChatGPT has captured the imagination and the internet, but does ChatGPT truly know everything, is it truly AGI? To answer some of these questions and to formulate what testing a conversational agent trained on a large language model would look like, we had chatGPT take the wit test where we asked it riddles. Find out what it said. 
Quick introduction to Large Language Models, how they are trained and what can be improved.
Slides and examples of the test associated with ChatGPT.
Discussion of why ChatGPT fails with understanding contexts, doesn't do well at verbal math, doesn't know what a venn diagram is and has never heard an egg crack and what it means for the next generation of a conversational AI model.
We will go deeper into how it perceives metaphorical language and convoluted relationships, we will also explore how the performance compares to that in the audience to have a good sense of what AGI means.

Quick introduction to Large Language Models, how they are trained and what can be improved.
Slides and examples of the test associated with ChatGPT.
Discussion of why ChatGPT fails with understanding contexts, doesn't do well at verbal math, doesn't know what a venn diagram is and has never heard an egg crack and what it means for the next generation of a conversational AI model.

Aroma Rodrigues

I talk to ChatGPT about things

MLOPs are a collection of practices that enable companies to build, train, deploy, scale, and operate models in production. Model serving is one of the main MLOps tasks. There are multiple ways of running models in production these days: from open-source solutions to enterprise offerings. However, a custom-built serving solution in Python is the most flexible option and can evolve together with your company's needs. 

The quality of an ML service is defined by its speed, accuracy, and ability to deal with the load. FastAPI is faster than its predecessors. Also, being a part of the Python ecosystem, it supports all the main ML frameworks. Moreover, it supports the OpenAPI standard out of the box and makes data validation much easier. All of this and its concurrency capability make it a great choice for running ML models in production.

Monika Venčkauskaitė

ML model serving and monitoring with FastAPI

Apple’s blocking of the IDFA identifier has made it difficult to attribute Apple users to marketing channels. This poses a big problem to many as it is harder to trace which channels are most effective in driving user-growth. It marks the first step towards the industry having to adapt to a more privacy-centric world where it is harder to track user-level data. 

In this talk, Avision will discuss how Mettle have reduced their reliance on user-level data by building a channel-level custom attribution model. This model enabled us to drive efficiencies in re-directing our spend on our strongest channels, leading to higher acquisition at lower cost. 

Some of the things we will deep-dive into is why this we use statsmodels instead of scikit-learn; how we benchmark our model’s appropriateness in the absence of an actual target; quickly servicing the insights to drive business-decisions as fast as possible; and then putting it into production via an Apache Airflow and BigQuery.

Avision Ho

Market attribution in an increasingly privacy-centric industry

Have you ever wanted to share Jupyter Notebook with non-technical users? Mercury is a new way to add widgets to a notebook and share it with non-programmers. You can easily build a dashboard, reports, web app, interactive slides, or REST API. Mercury allows you to add widgets to Jupyter Notebook. After the widget change, all cells below the widget are reexetuted with a new widget value. This simple execution model allows converting any Jupyter Notebook into an interactive web application. You can easily create a dashboard or presentation (slides in presentations can be recomputed during the show with values provided with widgets). What is more, Mercury allows schedule automatic execution easily. The framework has a built-in authentication module so that notebooks can be shared publicly or restricted with a password. Mercury is an open-source framework.

Have you ever wanted to share Jupyter Notebook with non-technical users? Mercury is a new way to add widgets to a notebook and share it with non-programmers. You can easily build a dashboard, reports, web app, interactive slides, or REST API. 
Mercury allows you to add widgets to Jupyter Notebook. After the widget change, all cells below the widget are reexetuted with a new widget value. This simple execution model allows converting any Jupyter Notebook into an interactive web application. You can easily create a dashboard or presentation (slides in presentations can be recomputed during the show with values provided with widgets). What is more, Mercury allows schedule automatic execution easily. The framework has a built-in authentication module so that notebooks can be shared publicly or restricted with a password. Mercury is an open-source framework.

Piotr Płoński

Aleksandra Płońska

Mercury widgets - a new way to make interactive webapp from Jupyter Notebook

Domain Driven Design (DDD) and Infrastructure from Code (IfC) are two powerful approaches to building software. DDD helps developers create flexible, scalable applications and with IfC they can be seamlessly deployed to the cloud. By combining these two approaches, we can create a layered architecture where IfC is just another layer in a DDD app. 
To illustrate how we can achieve this, I’ll show an example of an app I developed using DDD principles. To make it work with IfC, I needed to add a configuration layer and use a special Python syntax for the service layer, which enables the IfC engine to compile it. The other layers don't even know that they're running in the cloud, which makes it easy to maintain the application and add new features. This talk will provide insights into how you can leverage the power of DDD and IfC to create robust, scalable, and flexible software applications, and how to incorporate IfC as another layer in your DDD architecture.

Sources about Infrastructure from Code:
1. Jeremy Daly at re:Invent 2022 https://www.youtube.com/watch?v=RmwKBPCo7o4
2. Asher Sterkin at PyCon France 2023 https://www.youtube.com/watch?v=YB0UhznStlg

Details of the AWS credentials management task: https://medium.com/@barbara.toporowska/iam-credentials-janitor-fa0ca3337284

Barbara Toporowska

Domain Driven Design Meets Infractucture from Code: An AWS Credentials Management Case Study

Understanding software architecture and how the data flows within software components is a vital step toward building and maintaining software systems. Architecture diagrams help enable this through digital graphical designs mixed with human-computer interaction. Furthermore, these visualizations not only help system architects, but also developers, project managers, and even customers. The complexity in designing them arises not only from the fact that such systems are an intangible conceptual entity, but also, most importantly, that they are ever-evolving.

While we are searching for life on Mars, our software diagrams remain manual and lifeless. Imagine a life where you update the code for your software, and the architecture view gets updated automatically and is ready to be interacted with. Let's use Graph Databases, pandas, and Python to add life to them and make them interactive.

Before I dive into how to make "auto-generative software architecture visualizations," let's first see if you have been in any of the following situations:

1. You often wonder before you go to sleep, "A short time ago in a galaxy far, far away, I updated my team’s arch. diagram. So why is it stale… again?"
2. Your team grew multifold over the pandemic and so did your components and ownership. Has it become harder to onboard new engineers due to the ever-changing arch. diagrams, dependencies, and owners? 
3. Your new release didn't go as planned because something broke in your new code. Wouldn’t it help if you had a holistic and interactive view to navigate between components while understanding how they communicate with each other? When was the last time you used your arch. diagram as a debugging tool?
4. You managed to draw the data-flow diagram and you thought “this is majestic work right here!” But the feedback you received was, "it’s too detailed" or "it’s not detailed enough." While you’re trying to figure out the fine line between details, your hairline is starting to show! 

If you’ve fallen victim to any of these scenarios, behold! Your prayers have been answered! 

In the first half of the talk, we’ll describe an approach that can automatically identify the software design and data flows within the system. To achieve this, we use algorithmic scrapers and metadata profiling techniques that integrate with distributed trace, code structure, language dependencies, contribution ownerships, and other sources to avoid the toil of manual updates of software architecture diagrams. 

In the second part of the talk, we will dive deeper into the details of how to use Python data engineering libraries to enrich the collected data and process it to store it in the graph database. The graph structure not only signifies the relationship between components in a real-world manner, but also helps in generating multiple views of the software architecture in an easy and comprehensible fashion. Now, instead of the architecture and data flow diagram being a static JPEG, those auto-generative views can be compiled into interactive and immersive UIs.

My team built this tool for our organization, Bloomberg, but the approach we took is germane to all tech organizations. We want to share the challenges and lessons we learned during our journey to help you build a similar tool for your organization because why not “Code More, Draw Less”?!

Yash Saboo

Kang Min Bae

Code More, Draw Less: Auto-Generate Software Architecture Visualizations ft. Graph DBs, pandas & Python

The importance of enterprise architecture patterns is all well-known and applicable to varied types of tasks. Thinking about the architecture from the beginning of the journey is crucial to have a maintainable, therefore testable, and flexible code base. In We are going to explore the Ports and Adapters(Hexagonal) pattern by showing a simple web app using Repository, Unit of Work, and Services(Use Cases) patterns tied together with Dependency Injection. All those patterns are quite famous in other languages but they are relatively new for the Python ecosystem, which is a crucial missing part.

In nearly all web applications and Python tutorials we are starting from installing a web framework, and database server, the next step is to build database models and then use ORM, etc.
But wait, there is a problem with this classical approach, we lose the core business domain discussions - so-called core domain models just get lost inside some classes and functions. How about changing and reverting our approach? How about first starting by thinking, modeling our business, and core domain, and then testing it properly? Afterward, how about adding an abstraction layer on the database, then adding another abstraction on actual services, and use cases? But wait, how we are going to manage all transactional usage - okay let's add another layer with the Unit of Work pattern to manage our work as units. Sounds cryptic? Here is a step-by-step guide to starting our project:
* We are going to start with domain modeling and adding tests for our domain models
* The database layer will be abstracted using a Repository pattern
* The database transactions will be managed by the Unit of Work pattern
* The business logic actions were encapsulated in the Use Cases

The question can arise: where are our web framework and database server?
Answer: good architecture lets us defer those choices until the end. Because the web framework and the database server are details for our business/core application itself. Web framework will be considered as an entry point for our application and the database layer will be encapsulated using SQLAlchemy ORM, but still, ORM itself is hidden behind Repository and UoW patterns. This allows us to change the ORM library if there will be any need in the future.

The most important part here is to understand how we are going to build our application using Ports and Adapters(Hexagonal) pattern and all aforementioned patterns will be divided into Ports(using abstract base classes) and Adapters(the actual implementations), we can think about this as a contract between our actual implementations and abstractions.

Shahriyar Rzayev

Building Hexagonal Python Services

There are 2 most commonly used job schedulers in the Python world: Celery and Dramatiq. Neither of them supports use of Asyncio natively, which can significantly leverage performance of your application.

In this talk, we’ll discuss how you can use Repid to process large quantities of I/O bound tasks. We’ll then dive into the most useful features of the library that will provide you with the perfect framework to get the job done.

After this talk, you will be inspired to unleash the power of Asyncio in your message-driven systems!

Aleksandr Sulimov

Repid: new job scheduler with Asyncio in mind

The expectations on developers as well as their self-image has changed many times since the dawn of IT. Their role within an organization today is very different from the past - and more changes are on the horizon.
This talk will provide an overview based on experience with small to large enterprises over the last 30 years. What skills should a developer prepare today, in order to contribute to an organization in the future? What will development work look like in the future?

Robert Hoffmann

The role and skills of the developer: Past and Future

In a multi-region company it is not uncommon to encounter difficulties managing users, especially if they use a few products. 
I will share the process of creating a platform, including the challenges we faced and lessons learned from building it twice until we finally accomplished our goals.
How we used IDP to enforce security settings, how we migrated the users and how we used it to create new revenue streams.

In a multi-product company it is not uncommon to encounter difficulties managing users: each user has a unique identity, password, and configuration across various regions and products. The customers struggle to keep track of multiple login credentials and manage their users, while engineers have to duplicate code with custom adjustments to each product; Furthermore, cross-sells are less efficient.

A platform can assist to solve those problems, improve security, increase developer efficiency and enhance customer experience.

I will share the process of creating a platform, including the challenges we faced and lessons learned from building it twice until we finally accomplished our goals.
How we used IDP to enforce security settings, how we migrated the users and how we used it to create new revenue streams.

Hila Israeli

One Platform for All: A Revolution for Customers, Developers, and Sales

I'm Customer Success Engineer at JetBrains. Earlier, I led the development of Scala and Kotlin plugins for IntelliJ IDEA. But now, I'm covering the whole scale of our products, including PyCharm.

And here at PyCon LT as I'm ready to listen and discuss your current PyCharm problems. We will exchange for top-level problems and then go deeper into details if have time. Of course, it would be great if you share what you like (new UI? :) ).

Alexander Podkhalyuzin

PyCharm, let's discuss your problems

We improved the performance, reduced costs, and increased user engagement for the largest B2B pharmaceutical marketplace. Using Python Django, we made the website 4x faster, reduced infrastructure costs by 50%, and decreased bugs by 80%. We also enabled a full CI/CD process, implemented platform alerting and monitoring, and increased delivery rates by 2x. Our team's experience in software development allowed us to revolutionize the marketplace's web development experience, resulting in a 38% growth in organic traffic and a significant increase in website traffic, conversions, and revenue. We also started with a platform built on Next.js that had a performance score of 25, but ended up with a platform that achieved a perfect 100 score on Google. Average platform operation times reduced from 1.2s to 250 ms, while operating on avg 3m requests month.

Tadas Pikutis

Largest B2B pharma marketplace online: 7 years effors redone in a year thanks to python

Python web frameworks such as FastAPI, Flask, Quartz, Tornado, and Twisted are important for developing high-performance web applications and for their contributions to the web ecosystem. However, they may also present certain bottlenecks, either due to their synchronous nature or due to the usage of the Python runtime. These limitations can often arise from their reliance on *SGIs, which can limit the speed and performance of the web application. This is where Robyn comes in. Robyn aims to achieve near-native Rust throughput while still allowing developers to write code in Python. This means that developers can enjoy the benefits of high-performance web applications while still using the language they are comfortable with. In this talk, we will delve into Robyn and explore how it can improve web application performance. We will also discuss the development of Robyn and examine the evolution of a project from ideation to community support. We will examine the challenges and opportunities that arose during the development and how they were addressed.

With the rise of Rust bindings being used in the Python ecosystem, we know that throughput efficiency is one of the top priority items in the Python ecosystem.

Inspired by the extensibility and ease of use of the Python Web ecosystem and the increase of performance by using Rust as a core, Robyn was created. 

Robyn is one of the fastest Python web frameworks in the current Python web ecosystem. With a runtime written in Rust, Robyn achieves near-native rust performance while still having the ease of writing Python code. 

This talk will focus on the increased involvement of Rust in the Python ecosystem. It will also demonstrate why Robyn was created, the technical decisions behind Robyn, the increased performance by using the Rust runtime, how to use Robyn to develop web apps, and most importantly, how the community is helping Robyn grow!

I will briefly demonstrate my experience and challenges of building a community around the project and how it allowed Robyn to ensure a smooth sail even in turbulent situations. I shall also share my future plans for Robyn.

Sanskar Jethi

Robyn: A fast async Python web framework with a Rust runtime

Domain Driven Design (DDD) is a design approach that puts the business model as the core ground modeling the system’s design and closes the gap between the business logic and code. Amazon Web Services (AWS) empowers DDD with tons of services that can boost your architecture with on-demand databases, message buses, and cloud computing units. And everything connects with a cherry on top - Python’s microservices on serverless AWS resources. In this talk, I will present a use case of how we augmented the client’s team with our experts and helped build the cloud platform for smart metering of IoT devices. The main focus will be put not on theory but rather on showing the technical details and feedback on both: AWS and Python and how they work together to make thousands of devices' data ingestion and data analysis possible for smart buildings.
The outline:
- Short intro about the project
- DDD and how it looks in practice
- Python’s role in this project and tactical patterns when building microservices
- Conclusions what really worked out and what could be avoided
- Q&A

Unleashing the Power of Domain Driven Design and AWS with Python microservices

In the usual scenario, Django is served using WSGI server (gunicorn, uWSGI are the most popular) that work in pre-fork synchronous model, 
hence you will need multiple workers and have a “copy” of your application for each worker. 

This consumes a lot of memory and drags scalability issues along with it,
since one worker is busy responding to only one request at a time.

Green threads are one solution to NOT rewrite your whole codebase in order to go async without bumping into threads and the notorious GIL.

For the demo a simple Django concurrent application will be used and some benchmarks from a load test.

Django usually is considered slow and bad at doing concurrency because of its synchronous by default nature, but what if you could make it “async” without rewriting all of your codebase? Ever heard of green threads?

Anas El Amraoui

How to scale old Django apps for free

[Streamlit](https://streamlit.io/) is a popular framework for interactive web-based data apps in Python. However, there are some cases where users want to run their apps offline or without sending sensitive data to remote servers. To address these concerns, we introduce '[stlite](https://github.com/whitphx/stlite)': a WebAssembly port of Streamlit. It provides offline capability, data privacy, scalability, and multi-platform portability including desktop app packaging, while preserving Streamlit's original features, such as Python productivity and its rich ecosystem.

after a short intro of Streamlit, we will review stlite in the context of the recent emergence of various Wasm-based Python frameworks such as PyScript, and show you what's possible with stlite.
We will also look at its internals from a technical point of view, which may inspire you with ideas on how to make use of Pyodide and how to transform Python frameworks for the Pyodide/Wasm runtime.

You can try out stlite online: https://edit.share.stlite.net/

Yuichiro Tachibana

Streamlit meets WebAssembly - stlite

Mozilla has been promoting WASM for years, on the other hand, HTMX is gaining attraction. Question is, do we want more frontend or more backend? Do we still need to write JavaScripts?

In the first half of the talk we would explore the history of WASM and the Iodide project, what they enable and the closing of the Iodide project. Then we will talk about the rise of the Pyodide project and what this project enables - including another popular framework - PyScript. There will be some quick code demo of both Pyodide and PyScript.

In the second half of the talk, we will switch our attention to HTMX, what's the idea behind it and how it can be used to access AJAX, CSS Transitions, WebSockets and Server-Sent Events directly in HTML. There will also be some code demos of how to use HTMX, especially using it together with Django.

In the last part of the talk, there will be a conclusion, do we want more backend or more frontend? And most importantly, will web developers ever need to write JavaScript anymore?

Talks

Keynote

PyData

Python

Web development

Workshops

PyData

Python

Web development