SLO Burn—Reducing Alert Fatigue and Maintenance Cost in Systems of Any Size

Monday, October 29, 2018 - 11:00 am11:45 am

Jamie Wilkinson, Google

Abstract: 

Based on a true story.

As systems grow, they get more components, and more ways to fail. The alerts of the last system's design can slowly "boil the frog", and all of a sudden the SRE team finds they have no time left to address scaling problems because they're constantly firefighting. Alert fatigue sets in and the team will burn out.

Naturally maintenance work will always increase as the system itself grows. To make alerting sustainable, instead of on cause, only page on symptom, and even then by declaring what the acceptable threshold of symptom is -- also known as the SLO (and it's complement, the error budget).

Even at Google scale, many teams are yet to implement the change in their monitoring to realise SLO based alerts. But systems don't need to be the size of a planet to benefit from these patterns.

Whether you're oncall for 10 machines or 10 datacenters, in this talk a well rested champion of work/life balance will show you how to select service objectives, then to construct robust and low maintenance alerting rules using Prometheus for a live demonstration. We'll also discuss the tooling required to help make such a system retain observability in the absence of noisy caused-based alerts, now they're not telling you exactly which components are failing.

Jamie Wilkinson, Google

Jamie Wilkinson is a Site Reliability Engineer at Google. Contributing author to the "SRE Book", he has presented on contemporary topics at prominent conferences such as linux.conf.au, Monitorama, PuppetConf, Velocity, and SRECon. His interests began in monitoring and automation of small installations, but continues with human factors in automation and systems maintenance on large systems. Despite over 15 years in the industry, he is still trying to automate himself out of a job.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@conference {221696,
author = {Jamie Wilkinson},
title = {{SLO} Burn{\textemdash}Reducing Alert Fatigue and Maintenance Cost in Systems of Any Size},
year = {2018},
address = {Nashville, TN},
publisher = {{USENIX} Association},
}