Combining Buffered I/O and Direct I/O in Distributed File Systems


Yingjin Qian, Data Direct Networks; Marc-André Vef, Johannes Gutenberg University Mainz; Patrick Farrell and Andreas Dilger, Whamcloud Inc.; Xi Li and Shuichi Ihara, Data Direct Networks; Yinjin Fu, Sun Yat-Sen University; Wei Xue, Tsinghua University and Qinghai University; André Brinkmann, Johannes Gutenberg University Mainz


Direct I/O allows I/O requests to bypass the Linux page cache and was introduced over 20 years ago as an alternative to the default buffered I/O mode. However, high-performance computing (HPC) applications still mostly rely on buffered I/O, even if direct I/O could perform better in a given situation. This is because users tend to use the I/O mode they are most familiar with. Moreover, with complex distributed file systems and applications, it is often unclear which I/O mode to use.

In this paper, we show under which conditions both I/O modes are beneficial and present a new transparent approach that dynamically switches to each I/O mode within the file system. Its decision is based not only on the I/O size but also on file lock contention and memory constraints. We exemplary implemented our design into the Lustre client and server and extended it with additional features, e.g., delayed allocation. Under various conditions and real-world workloads, our approach achieved up to 3× higher throughput than the original Lustre and outperformed other distributed file systems that include varying degrees of direct I/O support by up to 13×.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {294777,
author = {Yingjin Qian and Marc-Andr{\'e} Vef and Patrick Farrell and Andreas Dilger and Xi Li and Shuichi Ihara and Yinjin Fu and Wei Xue and Andre Brinkmann},
title = {Combining Buffered {I/O} and Direct {I/O} in Distributed File Systems},
booktitle = {22nd USENIX Conference on File and Storage Technologies (FAST 24)},
year = {2024},
isbn = {978-1-939133-38-0},
address = {Santa Clara, CA},
pages = {17--33},
url = {},
publisher = {USENIX Association},
month = feb

Presentation Video