OpenEC: Toward Unified and Configurable Erasure Coding Management in Distributed Storage Systems

Authors: 

Xiaolu Li, Runhui Li, and Patrick P. C. Lee, The Chinese University of Hong Kong; Yuchong Hu, Huazhong University of Science and Technology

Abstract: 

Erasure coding becomes a practical redundancy technique for distributed storage systems to achieve fault tolerance with low storage overhead. Given its popularity, research studies have proposed theoretically proven erasure codes or efficient repair algorithms to make erasure coding more viable. However, integrating new erasure coding solutions into existing distributed storage systems is a challenging task and requires non-trivial re-engineering of the underlying storage workflows. We present OpenEC, a unified and configurable framework for readily deploying a variety of erasure coding solutions into existing distributed storage systems. OpenEC decouples erasure coding management from the storage workflows of distributed storage systems, and provides erasure coding designers with configurable controls of erasure coding operations through a directed-acyclic-graph-based programming abstraction. We prototype OpenEC on two versions of HDFS with limited code modifications. Experiments on a local cluster and Amazon EC2 show that OpenEC preserves both the operational performance and the properties of erasure coding solutions; OpenEC can also automatically optimize erasure coding operations to improve repair performance.

FAST '19 Open Access Sponsored by NetApp

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {227806,
author = {Xiaolu Li and Runhui Li and Patrick P. C. Lee and Yuchong Hu},
title = {OpenEC: Toward Unified and Configurable Erasure Coding Management in Distributed Storage Systems},
booktitle = {17th {USENIX} Conference on File and Storage Technologies ({FAST} 19)},
year = {2019},
isbn = {978-1-931971-48-5},
address = {Boston, MA},
pages = {331--344},
url = {https://www.usenix.org/conference/fast19/presentation/li},
publisher = {{USENIX} Association},
}