Want to Solve Over-Monitoring and Alert Fatigue? Create the Right Incentives!

Wednesday, 30 August, 2017 - 12:0012:30

Kishore Jalleda, Yahoo, Inc.


Telemetry monitors and their (constant) beeping is a pretty common sight in hospitals. I saw these at the NICU where my twins were being cared for after being born prematurely. My wife and I used to freak out every time one of these went off. Unlike a missed alarm that said your site's down, failing to act on an alarm at a hospital can have much more critical consequences; in 2010 at a hospital in Massachusetts, a patient's death was directly linked to telemetry monitoring after alarms signaling a critical event went unnoticed by 10 nurses.

I attempted to solve this problem when I joined Zynga (in 2013) as the head of SRE. I will go over our failed attempts including filtering the noise, adding heads, building more tools, etc. Will also cover how I came up with an initiative called "clean room" as a way to incentivize engineering teams to keep the noise levels low. Finally, go over some of the tactics that worked (and ones that didn't).

Most people I spoke to about "clean room" almost always walked away having learned something (some have said it's common sense). Share, learn, ask questions, participate - I'll try to make it fun!

Kishore Jalleda, Yahoo, Inc.

Kishore Jalleda is currently the head of production engineering in the Americas in Yahoo’s Publisher Products unit, which includes many popular destinations like Yahoo.com, Yahoo Finance, Yahoo Sports, Yahoo News and Flurry. Previously, Kishore was the head of SRE at Zynga and worked at IMVU, one of the pioneers of continuous delivery (with co-founder Eric Ries, the author of Lean Startup).

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@conference {205470,
author = {Kishore Jalleda},
title = {Want to Solve {Over-Monitoring} and Alert Fatigue? Create the Right Incentives!},
year = {2017},
address = {Dublin},
publisher = {USENIX Association},
month = aug

Presentation Video 

Presentation Audio