Build Your Own Observability Tool from Scratch for AI Agents

In this hands-on workshop, we will learn how to develop a custom observability solution tailored for AI agents. From logs and metrics to guardrail checks and cost tracking, we’ll discover how to monitor crucial elements like accuracy, latency, and environmental impact. Walk away with your own tool and best practices to ensure AI agents are both well-observed and well-behaved.

Basics of AI Agents, Python and related tools

AI agents can exhibit complex behaviors, including inconsistent accuracy, and unexpected spikes in resource usage. Traditional monitoring tools often fail to capture these nuances effectively. In this workshop, we will build an observability tool from scratch using Python, integrating techniques like advanced logging, metrics tracking, guardrail checks, and real-time dashboards. We will also address carbon emissions and cost visibility, highlighting the importance of responsible AI development. By the end, you’ll have a solid understanding of how to create a fully customizable solution that scales and adapts to your AI workflows.