Title: Towards a Federated Global-Scale Storage System

Presenter: Daniel Ellard (Network Appliance)

Co-authors: Craig Everhart (Network Appliance), Renu Tewari (IBM
Almaden), Manoj Naik (IBM Almaden)

Text:

The requirements for enterprise-level storage server capacity and
performance are growing more rapidly than monolithic servers can
scale.  At the same time, the users of these systems often want to
view their storage resources as an organization-wide aggregate rather
than partitioned among their servers.  The "obvious" solution to these
problems is to knit together servers into clusters that provide the
illusion of a monolithic server -- but many cluster architectures
bring their own challenges.  The first of these is compatibility with
standard storage protocols.  The second is the inability of most
cluster protocols to coordinate the activities of servers that are
dispersed geographically or administered by different organizations.

Our goal is to unify islands of data into a single federated data
space.  From a user's point of view, such a system provides
multi-protocol clients with what appears to be a single directory tree
but which is actually implemented as a federated global namespace that
makes a large set of volumes and datasets, possibly hosted on a
heterogeneous and geographically dispersed set of servers, appear to
the user as a single file system.  Behind the scenes, the volumes and
datasets may migrate from one location to another and may be
replicated in more than one location -- all transparently to the user.

From an administrator's point of view, such a system provides a way to
join a federation of servers and to share resources without
surrendering local control.  The challenge is to provide mechanisms
for federating the namespace and migration/replication policies
so that different parts of the system can be managed by different
sets of administrators with mutually exclusive (or perhaps competitive)
responsibilities and goals.

I will discuss on-going work at IBM (the Glamour project) and Network
Appliance (the Glitz project) to build such a system.  Much of the
work can be done by standard protocols and utilities (such as CIFS,
NFSv3, NFSv4, LDAP, DNS, and autofs), which means that unmodified
clients and legacy servers can be used, as-is, as components of the
system.  Some of the servers, however, run additional agents and/or
communicate over new protocols.  I will briefly describe these agents
and protocols, their properties, the new architectures they enable,
and our plans to release these protocols as open standards.