Towards Better Understanding of Black-box Auto-Tuning: A Comparative Analysis for Storage Systems


Zhen Cao, Stony Brook University; Vasily Tarasov, IBM Research - Almaden; Sachin Tiwari and Erez Zadok, Stony Brook University


Modern computer systems come with a large number of configurable parameters that control their behavior. Tuning system parameters can provide significant gains in performance but is challenging because of the immense number of configurations and complex, nonlinear system behavior. In recent years, several studies attempted to automate the tuning of system configurations; but they all applied only one or few optimization methods. In this paper, for the first time, we apply and then perform comparative analysis of multiple blackbox optimization techniques on storage systems, which are often the slowest components of computing systems. Our experiments were conducted on a parameter space consisting of nearly 25,000 unique configurations and over 450,000 data points. We compared these methods for their ability to find near-optimal configurations, convergence time, and instantaneous system throughput during auto-tuning. We found that optimal configurations differed by hardware, software, and workloads -- and that no one technique was superior to all others. Based on the results and domain expertise, we begin to explain the efficacy of these important automated blackbox optimization methods from a systems perspective.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

Presentation Audio

@inproceedings {215953,
author = {Zhen Cao and Vasily Tarasov and Sachin Tiwari and Erez Zadok},
title = {Towards Better Understanding of Black-box Auto-Tuning: A Comparative Analysis for Storage Systems},
booktitle = {2018 {USENIX} Annual Technical Conference ({USENIX} {ATC} 18)},
year = {2018},
isbn = {978-1-931971-44-7},
address = {Boston, MA},
pages = {893--907},
url = {},
publisher = {{USENIX} Association},