This tutorial is oriented toward administrators and developers who manage and
use HPC systems, and especially for those involved with storage systems in
these environments. Storage is often a critical part of the HPC infrastructure.
An important goal of the tutorial is to give the audience the foundation for
effectively comparing different storage system options, as well as a better
understanding of the systems they already have.
Cluster-based parallel storage technologies are used to manage millions of
files, thousands of concurrent jobs, and performance that scales from 10s to
100s of GB/sec. This tutorial will examine current state-of-the-art
high-performance file systems and the underlying technologies employed to
deliver scalable performance across a range of scientific and industrial
applications.
The tutorial starts with a look at storage devices and SSDs, in particular,
which are growing in importance in all storage systems. Next we look at how
a file system is put together, comparing and contrasting SAN file systems,
scale-out NAS, and object-based parallel file system architectures.
Topics
include scaling the data path, scaling metadata, fault tolerance, and
manageability. Specific systems are discussed, including Lustre, GPFS, PanFS,
HDFS (Hadoop File System), OpenStack, and the NFSv4.1 standard for parallel
I/O. We continue up the stack to discuss MPI-IO middleware that is often used
in large parallel programming environments for efficient I/O at scale.
Topics include:
Scaling the data path
Scaling metadata
Fault tolerance
Manageability
Specific systems are discussed, including Lustre, GPFS, PanFS, HDFS (Hadoop File System), OpenStack, and the NFSv4.1 standard for parallel I/O. We continue up the stack to discuss MPI-IO middleware that is often used in large parallel programming environments for efficient I/O at scale.
