HARDFS: Hardening HDFS with Selective and Lightweight Versioning

Authors: 

Thanh Do, Tyler Harter, and Yingchao Liu, University of Wisconsin—Madison; Haryadi S. Gunawi, University of Chicago; Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau, University of Wisconsin—Madison

Abstract: 

We harden the Hadoop Distributed File System (HDFS) against fail-silent (non fail-stop) behaviors that result from memory corruption and software bugs using a new approach: selective and lightweight versioning (SLEEVE). With this approach, actions performed by important subsystems of HDFS (e.g., namespace management) are checked by a second implementation of the subsystem that uses lightweight, approximate data structures. We show that HARDFS detects and recovers froma wide range of fail-silent behaviors caused by random bit flips, targeted corruptions, and real software bugs. In particular, HARDFS handles 90% of the fail-silentfaults that result from random memory corruption and correctly detects and recovers from 100% of 78 targeted corruptions and 5 real-world bugs. Moreover, it recovers orders of magnitude faster than full reboot by using micro-recovery. The extra protection in HARDFS incurs minimal performance and space overheads.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {180731,
author = {Thanh Do and Tyler Harter and Yingchao Liu and Haryadi S. Gunawi and Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau},
title = {{HARDFS}: Hardening {HDFS} with Selective and Lightweight Versioning},
booktitle = {11th USENIX Conference on File and Storage Technologies (FAST 13)},
year = {2013},
isbn = {978-1-931971-99-7},
address = {San Jose, CA},
pages = {105--118},
url = {https://www.usenix.org/conference/fast13/technical-sessions/presentation/do},
publisher = {USENIX Association},
month = feb,
}
Download
Do PDF
View the slides

Presentation Video 

Presentation Audio