AI in Production: Sampling, Labeling and Detecting Drift Before Your Customers Do
AI systems don't always fail when they go into production—they drift. Documents change, data distributions shift, vendors introduce new formats, and models behave differently on real-world inputs than they did in testing.
In this talk, we’ll build a practical evaluation loop on Google Cloud around a real-world AI document-processing pipeline. We’ll explore how to sample production data, create reliable ground truth through human labelling, evaluation, and detecting drift before it becomes a customer-facing problem.
We’ll walk through a live demo with Vertex AI, Document AI, BigQuery, Cloud Storage, and Cloud Run to create a continuous evaluation and feedback loop. The goal isn’t another accuracy dashboard. It’s a production feedback loop that tells you when your AI system is changing, why it changed, and what to do about it—before your customers do.
Akshata Mohanty is a Google Developer Expert in AI/ML and Head of Engineering at Tilla Technologies, Berlin. She founded Keis, a Singapore-based consultancy creating generative AI products for startups and venture studios. She has held senior roles at Rakuten, Nas.io, and the National University of Singapore, and speaks internationally on generative AI, MLOps, and cloud architecture.