Thursday, 15 October, 2026 - 09:00–09:45
Johan Mena and Albert Wang, Uber
Can your company survive a regional failover?
Company-wide failover drills answer that question, but they're operationally expensive to run, leaving long gaps for a bad deploy to quietly break your resilience. Staging won't save you either; you and I both know it's a lie since it can never match a complex system's real traffic and environment.
This is the story of how we built Muttloader at Uber to validate failover resilience directly in production.
We'll cover Muttloader's core traffic control mechanism for inducing higher load on services, the monitoring system that allows it to run safely in production, and what we learned from running tens of thousands of tests over five years of operation.



