Manta Storage System Internals

Manta is a system we have developed that applies the UNIX philosophy—small, well-defined tools and simple ways to combine them—to distributed computing on a multi-tenant object store. This new model of general-purpose compute not only unifies such disparate big data applications as indexing, log analysis, image processing, and video transcoding, but also carries with it a new set of design and implementation challenges.

In this talk, I'll discuss the architectural choices and implementation details, along with lessons learned as we created, deployed, and scaled the system. I'll pay particular attention to the aspects of running and automating the system where we've used Manta to operate and debug Manta, and how you can also use Manta to better operate your own services.

Mark Cavage is a software engineer at Joyent, where he works primarily on distributed systems supporting Joyent's cloud computing suite, and maintains several popular open source projects, such as restify and ldapjs. Prior to joining Joyent, Mark was a senior software engineer with Amazon Web Services, where he was primarily responsible for envisioning and launching the AWS Identity and Access Management product.