USENIX Conference Policies
Understanding and Dealing with Operator Mistakes in Internet Services
Operator mistakes are a significant source of unavailability in modern Internet services. In this paper, we first characterize these mistakes by performing an extensive set of experiments using human operators and a realistic three-tier auction service. The mistakes we observed range from software misconfiguration, to fault misdiagnosis, to incorrect software restarts. We next propose to validate operator actions before they are made visible to the rest of the system. We demonstrate how to accomplish this task via the creation of a validation environment that is an extension of the online system, where components can be validated using real workloads before they are migrated into the running service. We show that our prototype validation system can detect 66% of the operator mistakes that we have observed.
author = {Kiran Nagaraja and Fabio Oliveira and Ricardo Bianchini and Richard P. Martin and Thu D. Nguyen},
title = {Understanding and Dealing with Operator Mistakes in Internet Services},
booktitle = {6th Symposium on Operating Systems Design \& Implementation (OSDI 04)},
year = {2004},
address = {San Francisco, CA},
url = {https://www.usenix.org/conference/osdi-04/understanding-and-dealing-operator-mistakes-internet-services},
publisher = {USENIX Association},
month = dec
}