Reliability at Massive Scale: Lessons Learned at Facebook

As the Facebook Web site and platform grow to an ever larger scale, one of the most difficult challenges is running reliably while constantly changing our product. Over the years we have developed a number of principles around avoiding large failures while making frequent, small changes to our system. These principles have allowed us to run with a low rate of serious incidents, but they still do occur. I'll be walking through the details of a recent site outage to illustrate the way these principles work and how things can go wrong when they aren't followed.