Shripad J Nadgowda and Canturk Isci, Meta
At hyperscale, thousands of production experiments may need to exercise live downstream dependencies because staging cannot reproduce the full behavior, latency, and capacity profile of real systems. That realism catches regressions early, but it can also turn an experiment into a production incident with unclear ownership and a wide blast radius. This talk covers the operational safety architecture we built after those failures: dependency-by-dependency isolation decisions, deterministic task pinning for fast attribution, and experiment-local health signals that automatically cut traffic before shared services are harmed. Attendees will see the incidents and tradeoffs behind the design, including false positives, capacity overhead, and the cases where production fidelity is unavoidable. The goal is a practical framework for running high-fidelity pre-rollout testing without letting experiments become the most dangerous part of production.

Shripad Nadgowda is a Production Engineer at Meta, working on Meta Recommendation Systems (MRS) Infra powering Facebook, Instagram, and Threads. He has nearly two decades of experience across IBM Research, Intel, and Meta, spanning storage, virtualization, cloud computing, and production reliability. He has presented at KubeCon and various academic conferences, and at SREcon he will share practical lessons from operating experimentation platforms where reliability guardrails must keep pace with high-velocity change.

Canturk Isci is a Production Engineering Manager at Meta, working on Meta Recommendation Systems Infra and Agentic Reliability. He held prior technical leadership roles at IBM Research, VMware, Bloomberg and State Street. Canturk has a PhD in Computer Architecture from Princeton University, an MSc in VLSI Systems Design from University of Westminster, and a BS in Electronics Engineering from Bilkent University.

