Real-time data analytics is essential for powering modern applications like monitoring, personalization, search, and to some extend, RAG pipelines. However, building systems that can handle real-time ingestion, indexing, and retrieval at scale is no trivial task. This talk provides actionable insights into designing and maintaining such systems at scale using best practices.
basic of data engneering/analytics
Real-time data analytics systems are the backbone of modern applications, from monitoring and personalization to powering search experiences. However, building scalable systems that handle real-time ingestion, indexing, and retrieval efficiently can be daunting. This talk will explore the key components of such systems, including ingestion pipelines (Apache Kafka, Apache Pulsar etc), indexing layers (Elasticsearch, OpenSearch, FAISS, etc), and computation engines (Apache Flink, Apache Spark, custom Python-based solutions...). With the arrival of LLMs and Retrieval Augmented Generation (RAG), data indexation and retrieval become more critical than ever. We will address complex scenarios such as disaster recovery, data reconciliation, and ensuring low-latency performance at scale. Attendees will leave with actionable insights and best practices to design, implement, and maintain robust real-time analytics systems tailored to their needs.
I am a seasoned software engineer with an international footprint in Forbes 500 companies as well as VC-funded scale-ups in Europe and North America.
I have spent most of my career till now building resilient data and machine learning platforms. Most of them fail, some of them success but all of them give me lessons to learn from.
One of my success is being part of the team who built Komodo Health’s critical AI-fueled data analytic platform to actually save people. Our platform reduces the burden of disease research and reduce the cost of drug manufacturing and delivery.
Previously at Jellysmack, I worked on the core infrastructure team who scaled the data platform for a company who went from greenfield to unicorn startup in 2 years.
A part from that, I am a birds’ lover, an animal enthusiast, a day dreamer and a mediocre Judo player.