Metis: Robustly Tuning Tail Latencies of Cloud Systems

Tuning configurations is essential for operating modern cloud systems, but the difficulty arises from the cloud system’s diverse workloads, large system scale, and vast parameter space. The systems community has recently demonstrated the potential of predictive regression in minimizing the cost of searching for the optimal system configuration. However, we argue that cloud systems introduce challenges to the robustness of auto-tuning. First, system evaluation metrics such as tail latencies are typically sensitive to non-trivial noises. Second, while treating target systems as a black box promotes applicability, it complicates the process of selectively sample the unknown configuration-vs-performance space for modeling. To this end, Metis is an auto-tuning service used by several Microsoft services, and it implements customized Bayesian optimization to robustly improve auto-tuning: (1) the diagnostic model to find potential data outliers for re-sampling, and (2) a mixture of acquisition functions to balance exploitation, exploration and re-sampling. This paper uses the Bing Ads key- value store cluster as the running example – production results show that Metis has helped to lower the 99-percentile query lookup latency by more than 20.4%. In addition, Metis-tuned configurations outperform expert-tuned configurations, while reducing the tuning time from weeks to hours.