Combining SNIA Cloud, Tape, and Container Format Technologies for Long-Term Retention

Generating and collecting very large data sets is becoming a necessity in many domains that also need to keep that data for long periods. Examples include astronomy, genomics, medical records, photographic archives, video archives, and large-scale e-commerce. While this presents significant opportunities, a key challenge is providing economically scalable storage systems to efficiently store and preserve the data, as well as to enable search, access, and analytics on that data in the far future.

Both cloud and tape technologies are viable alternatives for storage of big data and SNIA supports their standardization. The SNIA Cloud Data Management Interface (CDMI) provides a standardized interface to create, retrieve, update, and delete objects in a cloud. The SNIA Linear Tape File System (LTFS) takes advantage of a new generation of tape hardware to provide efficient access to tape using standard, familiar system tools and interfaces. In addition, the SNIA Self-contained Information Retention Format (SIRF) defines a storage container for long term retention that will enable future applications to interpret stored data regardless of the application that originally produced it.

We'll present advantages and challenges in long-term retention of big data, as well as initial work on how to combine SIRF with LTFS and SIRF with CDMI to address some of those challenges. We will also describe an emerging SIRF specification as well as an implementation of SIRF for the cloud.

Dr. Fineberg is a Distinguished Technologist in the HP Storage Chief Technologist Office. In that role he leads technical strategy for big data, the cloud, and analytics storage. He helped develop the MPI2-I/O API and has over 20 years of experience in storage and high-performance computing I/O. Dr. Fineberg is currently co-chair of the Storage Networking Industry Association (SNIA) Long Term Retention Technical Working Group, and he is a member of the SNIA Analytics and Big Data Committee.