Repair Pipelining for Erasure-Coded Storage

Authors: 

Runhui Li, Xiaolu Li, Patrick P. C. Lee, and Qun Huang, The Chinese University of Hong Kong

Abstract: 

We propose repair pipelining, a technique that speeds up the repair performance in general erasure-coded storage. By pipelining the repair of failed data in small-size units across storage nodes, repair pipelining reduces the repair time to approximately the same as the normal read time to the same amount of data in homogeneous environments. We further extend repair pipelining for heterogeneous environments. We implement a repair pipelining prototype called ECPipe and integrate it as a middleware system into two open-source distributed storage systems HDFS and QFS. Experiments on a local testbed and Amazon EC2 show that repair pipelining significantly improves the performance of both degraded reads and full-node recovery over existing repair techniques.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {203207,
author = {Runhui Li and Xiaolu Li and Patrick P. C. Lee and Qun Huang},
title = {Repair Pipelining for Erasure-Coded Storage},
booktitle = {2017 {USENIX} Annual Technical Conference ({USENIX} {ATC} 17)},
year = {2017},
isbn = {978-1-931971-38-6},
address = {Santa Clara, CA},
pages = {567--579},
url = {https://www.usenix.org/conference/atc17/technical-sessions/presentation/li-runhui},
publisher = {{USENIX} Association},
month = jul,
}

Presentation Audio