Testing Configuration Changes in Context to Prevent Production Failures


Xudong Sun, Runxiang Cheng, Jianyan Chen, and Elaine Ang, University of Illinois at Urbana-Champaign; Owolabi Legunsen, Cornell University; Tianyin Xu, University of Illinois at Urbana-Champaign


Large-scale cloud services deploy hundreds of configuration changes to production systems daily. At such velocity, configuration changes have inevitably become prevalent causes of production failures. Existing misconfiguration detection and configuration validation techniques only check configuration values. These techniques cannot detect common types of failure-inducing configuration changes, such as those that cause code to fail or those that violate hidden constraints.

We present ctests, a new type of tests for detecting failure-inducing configuration changes to prevent production failures. The idea behind ctests is simple---connecting production system configurations to software tests so that configuration changes can be tested in the context of code affected by the changes. So, ctests can detect configuration changes that expose dormant software bugs and diverse misconfigurations.

We show how to generate ctests by transforming the many existing tests in mature systems. The key challenge that we address is the automated identification of test logic and oracles that can be reused in ctests. We generated thousands of ctests from the existing tests in five cloud systems.

Our results show that ctests are effective in detecting failure-inducing configuration changes before deployment. We evaluate ctests on real-world failure-inducing configuration changes, injected misconfigurations, and deployed configuration files from public Docker images. Ctests effectively detect real-world failure-inducing configuration changes and misconfigurations in the deployed files.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {258941,
author = {Xudong Sun and Runxiang Cheng and Jianyan Chen and Elaine Ang and Owolabi Legunsen and Tianyin Xu},
title = {Testing Configuration Changes in Context to Prevent Production Failures},
booktitle = {14th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 20)},
year = {2020},
isbn = {978-1-939133-19-9},
pages = {735--751},
url = {https://www.usenix.org/conference/osdi20/presentation/sun},
publisher = {{USENIX} Association},
month = nov,

Presentation Video