History-based Latency Prober Tuning

Tuesday, March 15, 2022 - 5:05 pm5:25 pm

Jeff Borwey, Google


Probers are an indispensable tool in monitoring production. When configured correctly, they offer high-fidelity insight into a system's performance and can provide fast detection and alerting for regressions. Performance, however, is not static and environments/deployments can behave radically different from one another. This talk will present some simple techniques for tuning latency prober alerts based on historical data. These techniques can increase sensitivity and reduce manual configuration toil while limiting false positives.

Jeff Borwey, Google

Jeff has been an SRE at Google for four years. Initially a BigQuery SRE, he now focuses on improving understanding and modeling of production performance more generally.

SREcon22 Americas Open Access Sponsored by Blameless

@conference {278152,
author = {Jeff Borwey},
title = {History-based Latency Prober Tuning},
year = {2022},
address = {San Francisco, CA},
publisher = {USENIX Association},
month = mar,

Presentation Video