On the Performance Variation in Modern Storage Stacks


Zhen Cao, Stony Brook University; Vasily Tarasov, IBM Research-Almaden; Hari Prasath Raman, Stony Brook University; Dean Hildebrand, IBM Research-Almaden; Erez Zadok, Stony Brook University


Ensuring stable performance for storage stacks is important, especially with the growth in popularity of hosted services where customers expect QoS guarantees. The same requirement arises from benchmarking settings as well. One would expect that repeated, carefully controlled experiments might yield nearly identical performance results—but we found otherwise. We therefore undertook a study to characterize the amount of variability in benchmarking modern storage stacks. In this paper we report on the techniques used and the results of this study. We conducted many experiments using several popular workloads, file systems, and storage devices—and varied many parameters across the entire storage stack. In over 25% of the sampled configurations, we uncovered variations higher than 10% in storage performance between runs. We analyzed these variations and found that there was no single root cause: it often changed with the workload, hardware, or software configuration in the storage stack. In several of those cases we were able to fix the cause of variation and reduce it to acceptable levels. We believe our observations in benchmarking will also shed some light on addressing stability issues in production systems.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {202254,
author = {Zhen Cao and Vasily Tarasov and Hari Prasath Raman and Dean Hildebrand and Erez Zadok},
title = {On the Performance Variation in Modern Storage Stacks},
booktitle = {15th USENIX Conference on File and Storage Technologies (FAST 17)},
year = {2017},
isbn = {978-1-931971-36-2},
address = {Santa Clara, CA},
pages = {329--344},
url = {https://www.usenix.org/conference/fast17/technical-sessions/presentation/cao},
publisher = {USENIX Association},
month = feb,

Presentation Audio