You are here

Home » HARDFS: Hardening HDFS with Selective and Lightweight Versioning

HARDFS: Hardening HDFS with Selective and Lightweight Versioning

We harden the Hadoop Distributed File System (HDFS) against fail-silent (non fail-stop) behaviors that result from memory corruption and software bugs using a new approach: selective and lightweight versioning (SLEEVE). With this approach, actions performed by important subsystems of HDFS (e.g., namespace management) are checked by a second implementation of the subsystem that uses lightweight, approximate data structures. We show that HARDFS detects and recovers froma wide range of fail-silent behaviors caused by random bit flips, targeted corruptions, and real software bugs. In particular, HARDFS handles 90% of the fail-silentfaults that result from random memory corruption and correctly detects and recovers from 100% of 78 targeted corruptions and 5 real-world bugs. Moreover, it recovers orders of magnitude faster than full reboot by using micro-recovery. The extra protection in HARDFS incurs minimal performance and space overheads.

Thanh Do, Tyler Harter, and Yingchao Liu, University of Wisconsin—Madison; Haryadi S. Gunawi, University of Chicago; Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau, University of Wisconsin—Madison

Open Access Content

Papers are restricted to registered attendees until the event begins. Once the event begins, the content becomes free and open to everyone. Journal articles are open to everyone upon publication.

Do PDF
View the slides
BibTeX
Text of BibTeX entry: 
@inproceedings {180731, author = {Thanh Do and Tyler Harter and Yingchao Liu and Haryadi S. Gunawi and Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau}, title = {HARDFS: Hardening HDFS with Selective and Lightweight Versioning}, booktitle = {Presented as part of the 11th USENIX Conference on File and Storage Technologies (FAST 13)}, year = {2013}, isbn = {978-1-931971-99-7}, address = {San Jose, CA}, pages = {105--118}, url = {https://www.usenix.org/conference/fast13/technical-sessions/presentation/do}, publisher = {USENIX}, } <br><a href="/biblio/export/bibtex/180731">Download</a>
Abstract: 

We harden the Hadoop Distributed File System (HDFS) against fail-silent (non fail-stop) behaviors that result from memory corruption and software bugs using a new approach: selective and lightweight versioning (SLEEVE). With this approach, actions performed by important subsystems of HDFS (e.g., namespace management) are checked by a second implementation of the subsystem that uses lightweight, approximate data structures. We show that HARDFS detects and recovers froma wide range of fail-silent behaviors caused by random bit flips, targeted corruptions, and real software bugs. In particular, HARDFS handles 90% of the fail-silentfaults that result from random memory corruption and correctly detects and recovers from 100% of 78 targeted corruptions and 5 real-world bugs. Moreover, it recovers orders of magnitude faster than full reboot by using micro-recovery. The extra protection in HARDFS incurs minimal performance and space overheads.

Presentation Video

Download Video

Presentation Audio

Open access to the papers is sponsored by USENIX.