From Research to Practice: Experiences Engineering a Production Metadata Database for a Scale Out File System
Authors:
Charles Johnson, Kimberly Keeton, and Charles B. Morrey III, HP Labs; Craig A. N. Soules, Natero; Alistair Veitch, Google; Stephen Bacon, Oskar Batuner, Marcelo Condotta, Hamilton Coutinho, Patrick J. Doyle, Rafael Eichelberger, Hugo Kiehl, Guilherme Magalhaes, James McEvoy, Padmanabhan Nagarajan, Patrick Osborne, Joaquim Souza, Andy Sparkes, Mike Spitzer, Sebastien Tandel, Lincoln Thomas, and Sebastian Zangaro, HP Storage
Abstract:
HP’s StoreAll with Express Query is a scalable commercial file archiving product that offers sophisticated file metadata management and search capabilities. A new REST API enables fast, efficient searching to find all files that meet a given set of metadata criteria and the ability to tag files with custom metadata fields. The product brings together two significant systems: a scale out file system and a metadata database based on LazyBase. In designing and building the combined product, we identified several real-world issues in using a pipelined database system in a distributed environment, and overcame several interesting design challenges that were not contemplated by the original research prototype. This paper highlights our experiences.
USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.
@inproceedings {179856,
author = {Charles Johnson and Kimberly Keeton and Charles B. Morrey III and Craig A. N. Soules and Alistair Veitch and Stephen Bacon and Oskar Batuner and Marcelo Condotta and Hamilton Coutinho and Patrick J. Doyle and Rafael Eichelberger and Hugo Kiehl and Guilherme Magalhaes and James McEvoy and Padmanabhan Nagarajan and Patrick Osborne and Joaquim Souza and Andy Sparkes and Mike Spitzer and Sebastien Tandel and Lincoln Thomas and Sebastian Zangaro},
title = {From Research to Practice: Experiences Engineering a Production Metadata Database for a Scale Out File System},
booktitle = {Proceedings of the 12th {USENIX} Conference on File and Storage Technologies ({FAST} 14)},
year = {2014},
isbn = {ISBN 978-1-931971-08-9},
address = {Santa Clara, CA},
pages = {191--198},
url = {https://www.usenix.org/conference/fast14/technical-sessions/presentation/johnson},
publisher = {{USENIX}},
}
connect with us