Crux Conception, M. Psych

Justinas Kuizinas

Uncle Data

Uncle Data session 2

Lightning talks

In this talk we will examine how Python programmers can be creative in the digital age. Whether it's in web development, data science or machine learning creativity is a skill that sets a good developer apart from a great one. Python as a language has been around for 32 years and it may seem like the old saying 'there is nothing new under the sun' is true since so much has been built with it. For this talk we'll look at why building off of the work of other people is essential for creativity and how open source is the thread that ties it all together.

Marlene Mhangami

Python and Creativity (An Explorers Guide)

Travis Oliphant

Leader talks

Marc Garcia

Ritchie Vink

Uncle Data session 3

From recommendation systems to LLM-based applications, vector search is a critical component of the modern AI workflow. Existing vector solutions are complicated to use, hard to maintain, and cost too much. LanceDB is a free open-source vector store that can perform low latency vector search on billion-scale vector datasets on a single node. LanceDB is powered by Lance format, a modern columnar data format for machine learning and data science. Compatible with pandas/polars/duckdb, Lance format supports vector index, predicate pushdown, and random access performance 2000x faster than parquet.

This talk will:
1. Introduce LanceDB and show some example workflows
2. Outline Lance format design and what makes it so fast
3. Review the Lance roadmap and ecosystem integrations

You can find Lance here: https://github.com/eto-ai/lance

Chang She

Serverless billion-scale vector search for AI applications

Have you ever wanted to share Jupyter Notebook with non-technical users? Mercury is a new way to add widgets to a notebook and share it with non-programmers. You can easily build a dashboard, reports, web app, interactive slides, or REST API. Mercury allows you to add widgets to Jupyter Notebook. After the widget change, all cells below the widget are reexetuted with a new widget value. This simple execution model allows converting any Jupyter Notebook into an interactive web application. You can easily create a dashboard or presentation (slides in presentations can be recomputed during the show with values provided with widgets). What is more, Mercury allows schedule automatic execution easily. The framework has a built-in authentication module so that notebooks can be shared publicly or restricted with a password. Mercury is an open-source framework.

Have you ever wanted to share Jupyter Notebook with non-technical users? Mercury is a new way to add widgets to a notebook and share it with non-programmers. You can easily build a dashboard, reports, web app, interactive slides, or REST API. 
Mercury allows you to add widgets to Jupyter Notebook. After the widget change, all cells below the widget are reexetuted with a new widget value. This simple execution model allows converting any Jupyter Notebook into an interactive web application. You can easily create a dashboard or presentation (slides in presentations can be recomputed during the show with values provided with widgets). What is more, Mercury allows schedule automatic execution easily. The framework has a built-in authentication module so that notebooks can be shared publicly or restricted with a password. Mercury is an open-source framework.

Piotr Płoński

Aleksandra Płońska

Mercury widgets - a new way to make interactive webapp from Jupyter Notebook

The talk will introduce Apache Airflow and its competitors. The main goal of the talk is to showcase how Airflow adapted to the ever-changing data space. Comparison and feature exploration would be oriented in tackling the simplest data extraction and transformation layers in different tools. The plan is to showcase extracting part from Database to S3, move to a database, and add a transformation layer to fill the data warehouse and run data quality checks. The comparison would be on speed, efficiency, integration of different tools and vendors (AWS, dbt Labs, Postgres db) and how it looks in the modern data engineering world (adjustments to more frequent refreshes, dataset awareness, community support), what's crucial, what's missing and how it might go in the future for all of these tools.

Tomas Peluritis

Is it the end for Apache Airflow?

More than 5 million people play Nordcurrent mobile games every month. The specificity of free-to-play games is that less than 10% of players make purchases. It is essential to retain paying players and keep them engaged as long as possible. To do that, we built a purchase prediction model.

We store data and make the most of feature engineering in Clickhouse. Apache Airflow orchestrates pipelines. Usually, we use CatBoost for Machine Learning. Pydantic and ClearML, on top of AWS S3, manage model files, training metrics, and configs. The quality in production is evaluated using dashboards in Apache Superset.

The architecture allows us to build fully reproducible ML pipelines. The learning process can be horizontally scaled to select the optimal hyperparameters. At the inference stage, you do not need to worry that the model was trained in some Jupiter Notebook, and it is unclear what to do if it suddenly breaks in a month.

Dima Savostyanov

How we predict purchases in mobile games

Most data transformations are written twice. In the field of feature engineering for Machine Learning, data scientists regularly have to build, manage, and iterate on batch jobs, then translate those jobs to a service setting to load data and make fresh predictions. At best, this process is an engineering headache. At worst, this can result in difficult-to-detect deltas between training and inference, complex code, and highly bespoke infrastructure. In this talk we discuss Hamilton, a lightweight open-source framework in python that enables data practitioners to cleanly and portably define dataflows. Hamilton places no restrictions on the nature of transformations, allowing data scientists to use their favorite python libraries. With Hamilton, you can run the same code in your airflow DAG for training as you would in your fastAPI service for inference, and get the same result.

In this talk, we present Hamilton, and talk about how it can enable data scientists to build highly portable dataflows that can run in a variety of different contexts. At a high level, we will discuss:
The paradigm Hamilton introduces, and how it is simplifies the process of building and maintaining feature engineering pipelines
How Hamilton can be used to help scale batch data preparation for training and inference
How the same hamilton code can be used in a web-service to prepare data and run live inference, with minimal changes

We will go over working code examples, making sure to connect with tooling people are familiar with (E.G. airflow, fastapi, metaflow, django).

Elijah ben Izzy

Portable Feature Engineering with Hamilton: Write Once, Run Everywhere

Writing maintainable data science code is a big topic, and different people have different opinions on the best ways to do it. Wouldn't it be nice if there was an opinionated framework to set some structure and help data scientists be more effective and ship their analysis and models to production faster?

In this workshop we present Kedro, an opinionated Python framework for creating reproducible, maintainable and modular data science code. We will also show how you can combine it with Polars, a new dataframe library backed by Arrow and Rust, for lightning fast data manipulation and exploratory data analysis.

In this workshop we present Kedro, an opinionated Python framework for creating reproducible, maintainable and modular data science code. We will also show how you can combine it with Polars, a new dataframe library backed by Arrow and Rust, for lightning fast data manipulation and exploratory data analysis.

Kedro is an open source (Apache 2.0) Python framework for maintainable data science that provides a series of project templates, a declarative data catalog, functionality to create function-based data pipelines, and a powerful visualization tool. It has a rich ecosystem of plugins and extensions and a thriving community.

Traditionally, Kedro has encouraged the use of pandas for data I/O and manipulation. In recent times, Polars has become increasingly popular thanks to its expressive API, its lazy evaluation system, its out of core capabilities, and its impressive performance.

The workshop will be hands on, and the outline is as follows:

1. The problem of maintainability in data science code
2. What is Kedro?
3. Quick data I/O with Polars
4. Introducing the Kedro catalog and the Jupyter integration
5. Creating pipelines in Kedro
6. More exploratory data analysis with Polars
7. Plots in Kedro Viz

Juan Luis Cano Rodríguez

Analyze your data at the speed of light with Polars and Kedro

We improved the performance, reduced costs, and increased user engagement for the largest B2B pharmaceutical marketplace. Using Python Django, we made the website 4x faster, reduced infrastructure costs by 50%, and decreased bugs by 80%. We also enabled a full CI/CD process, implemented platform alerting and monitoring, and increased delivery rates by 2x. Our team's experience in software development allowed us to revolutionize the marketplace's web development experience, resulting in a 38% growth in organic traffic and a significant increase in website traffic, conversions, and revenue. We also started with a platform built on Next.js that had a performance score of 25, but ended up with a platform that achieved a perfect 100 score on Google. Average platform operation times reduced from 1.2s to 250 ms, while operating on avg 3m requests month.

Tadas Pikutis

Largest B2B pharma marketplace online: 7 years effors redone in a year thanks to python

The importance of enterprise architecture patterns is all well-known and applicable to varied types of tasks. Thinking about the architecture from the beginning of the journey is crucial to have a maintainable, therefore testable, and flexible code base. In We are going to explore the Ports and Adapters(Hexagonal) pattern by showing a simple web app using Repository, Unit of Work, and Services(Use Cases) patterns tied together with Dependency Injection. All those patterns are quite famous in other languages but they are relatively new for the Python ecosystem, which is a crucial missing part.

In nearly all web applications and Python tutorials we are starting from installing a web framework, and database server, the next step is to build database models and then use ORM, etc.
But wait, there is a problem with this classical approach, we lose the core business domain discussions - so-called core domain models just get lost inside some classes and functions. How about changing and reverting our approach? How about first starting by thinking, modeling our business, and core domain, and then testing it properly? Afterward, how about adding an abstraction layer on the database, then adding another abstraction on actual services, and use cases? But wait, how we are going to manage all transactional usage - okay let's add another layer with the Unit of Work pattern to manage our work as units. Sounds cryptic? Here is a step-by-step guide to starting our project:
* We are going to start with domain modeling and adding tests for our domain models
* The database layer will be abstracted using a Repository pattern
* The database transactions will be managed by the Unit of Work pattern
* The business logic actions were encapsulated in the Use Cases

The question can arise: where are our web framework and database server?
Answer: good architecture lets us defer those choices until the end. Because the web framework and the database server are details for our business/core application itself. Web framework will be considered as an entry point for our application and the database layer will be encapsulated using SQLAlchemy ORM, but still, ORM itself is hidden behind Repository and UoW patterns. This allows us to change the ORM library if there will be any need in the future.

The most important part here is to understand how we are going to build our application using Ports and Adapters(Hexagonal) pattern and all aforementioned patterns will be divided into Ports(using abstract base classes) and Adapters(the actual implementations), we can think about this as a contract between our actual implementations and abstractions.

Shahriyar Rzayev

Building Hexagonal Python Services

Are you curious about how recommendation engines work? In this workshop, we'll dive deep into the world of TensorFlow Recommender Systems, exploring the fundamental concepts, techniques, and tools needed to build effective recommendation engines. We'll start with an overview of the different types of recommender systems, including collaborative filtering, content-based filtering, and hybrid models. We'll also explore evaluation metrics and learn how to measure the effectiveness of these models.
The workshop then shifts to hands-on exercises that allow you to build your own recommendation engine using TensorFlow. You'll learn how to prepare data, train the model, and make recommendations. Through guided examples, you'll gain a practical understanding of the end-to-end process of building a recommendation engine.

In today's digital world, personalized recommendations have become an essential part of user engagement and retention. With the abundance of data available, building effective recommendation systems can seem daunting. In this workshop, we will take a deep dive into the world of TensorFlow Recommender Systems and explore the techniques and tools necessary to build a high-performing recommendation engine.

We will start by understanding the fundamentals of recommender systems and how TensorFlow can be used to build them. We will then explore the different types of recommenders, including content-based, collaborative filtering, and hybrid models. We will also delve into the various evaluation metrics used to measure the effectiveness of recommender systems.

In the second half of the workshop, we will get hands-on experience with TensorFlow Recommender Systems. We will work through a real-world use case and learn how to prepare the data, build and train the model, and make recommendations. 

By the end of the workshop, attendees will have a solid understanding of the key concepts and tools required to build powerful recommender systems using TensorFlow. They will have the opportunity to apply what they have learned and create their own personalized recommendations. Whether you're a data scientist, engineer, or product manager, this workshop is the perfect opportunity to dive deeper into the world of TensorFlow Recommender Systems and take your skills to the next level.

Ashmi Banerjee

The Ultimate Matchmaker: Building Recommender Systems with TensorFlow

Building Streamlit apps that enable business users to explore data on their own is an excellent way to support data-driven decision-making in any organization. Pair this up with the animated transitions between the charts provided by the new, open-source library, ipyvizzu, and you have a self-service, interactive report or dashboard that makes it much easier for non-experts to make sense of complex data sets.

It's great when you can share the results of your analysis not only as a presentation but as something that non-data scientists can explore on their own, looking for insights and applying their business expertise to understand the significance of what they find.

With its accessibility for both creators and viewers, Streamlit offers a brilliant platform for data scientists to build and deploy data apps. Now, with the integration of [ipyvizzu](https://ipyvizzu.com) - a new, open-source data visualization tool focusing on animation and storytelling - you can quickly create and publish interactive, animated reports and dashboards on top of static or dynamic data sets and your models.

In this talk, one of the creators of ipyvizzu shows how their technology works within Streamlit and the advantages of using animation in self-service data exploration to help business stakeholders feel smarter and do a better job.

Peter Vidos

Let them explore! Building interactive, animated reports in Streamlit with ipyvizzu & a few lines of Python

Company presentations

Company presentations 2

Domain Driven Design (DDD) is a design approach that puts the business model as the core ground modeling the system’s design and closes the gap between the business logic and code. Amazon Web Services (AWS) empowers DDD with tons of services that can boost your architecture with on-demand databases, message buses, and cloud computing units. And everything connects with a cherry on top - Python’s microservices on serverless AWS resources. In this talk, I will present a use case of how we augmented the client’s team with our experts and helped build the cloud platform for smart metering of IoT devices. The main focus will be put not on theory but rather on showing the technical details and feedback on both: AWS and Python and how they work together to make thousands of devices' data ingestion and data analysis possible for smart buildings.
The outline:
- Short intro about the project
- DDD and how it looks in practice
- Python’s role in this project and tactical patterns when building microservices
- Conclusions what really worked out and what could be avoided
- Q&A

Unleashing the Power of Domain Driven Design and AWS with Python microservices

In a multi-region company it is not uncommon to encounter difficulties managing users, especially if they use a few products. 
I will share the process of creating a platform, including the challenges we faced and lessons learned from building it twice until we finally accomplished our goals.
How we used IDP to enforce security settings, how we migrated the users and how we used it to create new revenue streams.

In a multi-product company it is not uncommon to encounter difficulties managing users: each user has a unique identity, password, and configuration across various regions and products. The customers struggle to keep track of multiple login credentials and manage their users, while engineers have to duplicate code with custom adjustments to each product; Furthermore, cross-sells are less efficient.

A platform can assist to solve those problems, improve security, increase developer efficiency and enhance customer experience.

I will share the process of creating a platform, including the challenges we faced and lessons learned from building it twice until we finally accomplished our goals.
How we used IDP to enforce security settings, how we migrated the users and how we used it to create new revenue streams.

Hila Israeli

One Platform for All: A Revolution for Customers, Developers, and Sales

I'm Customer Success Engineer at JetBrains. Earlier, I led the development of Scala and Kotlin plugins for IntelliJ IDEA. But now, I'm covering the whole scale of our products, including PyCharm.

And here at PyCon LT as I'm ready to listen and discuss your current PyCharm problems. We will exchange for top-level problems and then go deeper into details if have time. Of course, it would be great if you share what you like (new UI? :) ).

Alexander Podkhalyuzin

PyCharm, let's discuss your problems

There are 2 most commonly used job schedulers in the Python world: Celery and Dramatiq. Neither of them supports use of Asyncio natively, which can significantly leverage performance of your application.

In this talk, we’ll discuss how you can use Repid to process large quantities of I/O bound tasks. We’ll then dive into the most useful features of the library that will provide you with the perfect framework to get the job done.

After this talk, you will be inspired to unleash the power of Asyncio in your message-driven systems!

Aleksandr Sulimov

Repid: new job scheduler with Asyncio in mind

Python web frameworks such as FastAPI, Flask, Quartz, Tornado, and Twisted are important for developing high-performance web applications and for their contributions to the web ecosystem. However, they may also present certain bottlenecks, either due to their synchronous nature or due to the usage of the Python runtime. These limitations can often arise from their reliance on *SGIs, which can limit the speed and performance of the web application. This is where Robyn comes in. Robyn aims to achieve near-native Rust throughput while still allowing developers to write code in Python. This means that developers can enjoy the benefits of high-performance web applications while still using the language they are comfortable with. In this talk, we will delve into Robyn and explore how it can improve web application performance. We will also discuss the development of Robyn and examine the evolution of a project from ideation to community support. We will examine the challenges and opportunities that arose during the development and how they were addressed.

With the rise of Rust bindings being used in the Python ecosystem, we know that throughput efficiency is one of the top priority items in the Python ecosystem.

Inspired by the extensibility and ease of use of the Python Web ecosystem and the increase of performance by using Rust as a core, Robyn was created. 

Robyn is one of the fastest Python web frameworks in the current Python web ecosystem. With a runtime written in Rust, Robyn achieves near-native rust performance while still having the ease of writing Python code. 

This talk will focus on the increased involvement of Rust in the Python ecosystem. It will also demonstrate why Robyn was created, the technical decisions behind Robyn, the increased performance by using the Rust runtime, how to use Robyn to develop web apps, and most importantly, how the community is helping Robyn grow!

I will briefly demonstrate my experience and challenges of building a community around the project and how it allowed Robyn to ensure a smooth sail even in turbulent situations. I shall also share my future plans for Robyn.

Sanskar Jethi

Robyn: A fast async Python web framework with a Rust runtime

The expectations on developers as well as their self-image has changed many times since the dawn of IT. Their role within an organization today is very different from the past - and more changes are on the horizon.
This talk will provide an overview based on experience with small to large enterprises over the last 30 years. What skills should a developer prepare today, in order to contribute to an organization in the future? What will development work look like in the future?

Robert Hoffmann

The role and skills of the developer: Past and Future

AI technologies brough the ability to construct the data in new light. Search engines learn to search not by keywords but by semantic information, the same goes for image or audio search. In this talk we dive into idea and realization how to construct Image/Text/Audio search from scratch, using existing AI pipelines.

We dive into usage of Pytorch library and existing pre-trained models, to construct Image/Text/Audio search from scratch, using existing AI pipelines. The investigation of paralelization of computations using numba (cuda) library will expand the speeup cases.

Linas Petkevičius

Similarity search in practice or how AI can help in everyday’s life

MLOPs are a collection of practices that enable companies to build, train, deploy, scale, and operate models in production. Model serving is one of the main MLOps tasks. There are multiple ways of running models in production these days: from open-source solutions to enterprise offerings. However, a custom-built serving solution in Python is the most flexible option and can evolve together with your company's needs. 

The quality of an ML service is defined by its speed, accuracy, and ability to deal with the load. FastAPI is faster than its predecessors. Also, being a part of the Python ecosystem, it supports all the main ML frameworks. Moreover, it supports the OpenAPI standard out of the box and makes data validation much easier. All of this and its concurrency capability make it a great choice for running ML models in production.