|
NSDI '04 — Abstract
Pp. 365–378 of the Proceedings
Explicit Control in the Batch-Aware Distributed File System
John Bent, Douglas Thain, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Miron Livny, University of Wisconsin, Madison
Abstract
We present the design, implementation, and evaluation
of the Batch-Aware Distributed File System (BAD-FS),
a system designed to orchestrate large, I/O-intensive
batch workloads on remote computing clusters distributed
across the wide area. BAD-FS consists of two novel components:
a storage layer that exposes control of traditionally
fixed policies such as caching, consistency, and
replication; and a scheduler that exploits this control as
necessary for different workloads. By extracting control
from the storage layer and placing it within an external
scheduler, BAD-FS manages both storage and computation
in a coordinated way while gracefully dealing with
cache consistency, fault-tolerance, and space management
issues in a workload-specific manner. Using both
microbenchmarks and real workloads, we demonstrate
the performance benefits of explicit control, delivering excellent
end-to-end performance across the wide-area.
- View the full text of this paper in PDF.
The Proceedings are published as a collective work, © 2004 by the USENIX Association. All Rights Reserved. Rights to individual papers remain with the author or the author's employer. Permission is granted for the noncommercial reproduction of the complete work for educational or research purposes. USENIX acknowledges all trademarks within this paper.
- If you need the latest Adobe Acrobat Reader, you can download it from Adobe's site.
|
Need help? Use our Contacts page.
