How Uber Validates Failover Resilience in Production

Thursday, 15 October, 2026 - 09:00–09:45

Johan Mena and Albert Wang, Uber

Can your company survive a regional failover?

Company-wide failover drills answer that question, but they're operationally expensive to run, leaving long gaps for a bad deploy to quietly break your resilience. Staging won't save you either; you and I both know it's a lie since it can never match a complex system's real traffic and environment.

This is the story of how we built Muttloader at Uber to validate failover resilience directly in production.

We'll cover Muttloader's core traffic control mechanism for inducing higher load on services, the monitoring system that allows it to run safely in production, and what we learned from running tens of thousands of tests over five years of operation.

Johan is a Staff Software Engineer at Uber on the Production Engineering team based in New York City. He focuses on building software that makes Uber safer and more efficient at scale.

Albert is a Senior Staff Technical Lead Manager based in San Francisco in the Production Engineering org. He works on a bit of everything in the platform engineering space from observability to scaling to reliability tooling.

Presentation Video