Jay Lees and Javier Martin Montull, Meta
In this talk, we'll delve into the world of continuous online A/B-test experiments, where hundreds of engineers are constantly iterating, tweaking, and tuning models that directly impact business outcomes. We'll discuss the challenges of instilling a reliability mindset into this rapidly changing environment, and share our experiences with implementing mechanisms for preventing, detecting, and quickly mitigating issues triggered by AI-experiments in large-scale systems.
Jay is a Production Engineer at Meta in the Monetization Infra & Ranking organization, where he focuses on Reliability problems across the Ads Delivery, Events & Reporting stacks. He spent the last 2 years working on Experiment Safety, making it easier and safer for engineers across the company to test and iterate on their ideas across both ML & non-ML applications.
Javier is a Production Engineering Manager at Meta, supporting teams in the Ads ML space. During his 7+ year career at Meta, Javier has been both an engineer and a manager and has always focused on the Ads infrastructure space. Prior to Meta, Javier spent 10 years working as a software engineer at the physics lab CERN.
author = {Jay Lees and Javier Martin Montull},
title = {Experimenting with {AI-Driven} Systems},
year = {2025},
address = {Dublin},
publisher = {USENIX Association},
month = oct
}
