Building Production-Ready On-Device Rewrite: Speed, Robustness, and Customer Impact

Speaker

Marat Saidov

Marat Saidov is a Senior Software Engineer at Applied Sciences Group, Microsoft. Based in Belgrade, Serbia. Previously improved Speech Recognition and Natural Language Understanding services at Alice Voice Assistant, Yandex. Besides that, he was an NLP Research Assistant at HSE University, Russia.

Abstract

I'll discuss building capabilities on top of on-device language models, with Rewrite as a case study – a publicly available paraphrasing skill that is widely used across Microsoft's products. I'll cover comprehensive data collection strategies, carefully designed adapters training and evaluation. I'll discuss the engineering challenges we faced: achieving target latency while maintaining quality, hardening the system against edge cases, and how to deliver the technology to partners and the lessons learnt.