Revitalizing the Forgotten On-Chip DMA to Expedite Data Movement in NVM-based Storage Systems


Jingbo Su, Jiahao Li, and Luofan Chen, University of Science and Technology of China; Cheng Li, University of Science and Technology of China and Anhui Province Key Laboratory of High Performance Computing; Kai Zhang and Liang Yang, SmartX; Sam H. Noh, UNIST & Virginia Tech; Yinlong Xu, University of Science and Technology of China and Anhui Province Key Laboratory of High Performance Computing


Data-intensive applications executing on NVM-based storage systems experience serious bottlenecks when moving data between DRAM and NVM. We advocate for the use of the long-existing but recently neglected on-chip DMA to expedite data movement with three contributions. First, we explore new latency-oriented optimization directions, driven by a comprehensive DMA study, to design a high-performance DMA module, which significantly lowers the I/O size threshold to observe benefits. Second, we propose a new data movement engine, Fastmove, that coordinates the use of the DMA along with the CPU with judicious scheduling and load splitting such that the DMA’s limitations are compensated, and the overall gains are maximized. Finally, with a general kernel-based design, simple APIs, and DAX file system integration, Fastmove allows applications to transparently exploit the DMA and its new features without code change. We run three data-intensive applications MySQL, GraphWalker, and Filebench atop NOVA, ext4-DAX, and XFS-DAX, with standard benchmarks like TPC-C, and popular graph algorithms like PageRank. Across single- and multi-socket settings, compared to the conventional CPU-only NVM accesses, Fastmove introduces to TPC-C with MySQL 1.13-2.16× speedups of peak throughput, reduces the average latency by 17.7-60.8%, and saves 37.1-68.9% CPU usage spent in data movement. It also shortens the execution time of graph algorithms with GraphWalker by 39.7-53.4%, and introduces 1.12-1.27× throughput speedups for Filebench.

FAST '23 Open Access Sponsored by

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

This content is available to:

@inproceedings {285754,
author = {Jingbo Su and Jiahao Li and Luofan Chen and Cheng Li and Kai Zhang and Liang Yang and Yinlong Xu},
title = {Revitalizing the Forgotten {On-Chip} {DMA} to Expedite Data Movement in {NVM-based} Storage Systems},
booktitle = {21st USENIX Conference on File and Storage Technologies (FAST 23)},
year = {2023},
isbn = {978-1-939133-32-8},
address = {Santa Clara, CA},
pages = {363--378},
url = {},
publisher = {USENIX Association},
month = feb

Presentation Video