Understanding Ephemeral Storage for Serverless Analytics

Authors: 

Ana Klimovic, Yawen Wang, and Christos Kozyrakis, Stanford University; Patrick Stuedi, Jonas Pfefferle, and Animesh Trivedi, IBM Research

Abstract: 

Serverless computing frameworks allow users to launch thousands of concurrent tasks with high elasticity and fine-grain resource billing without explicitly managing computing resources. While already successful for IoT and web microservices, there is increasing interest in leveraging serverless computing to run data-intensive jobs, such as interactive analytics. A key challenge in running analytics workloads on serverless platforms is enabling tasks in different execution stages to efficiently communicate data between each other via a shared data store. In this paper, we explore the suitability of different cloud storage services (e.g., object stores and distributed caches) as remote storage for serverless analytics. Our analysis leads to key insights to guide the design of an ephemeral cloud storage system, including the performance and cost efficiency of Flash storage for serverless application requirements and the need for a pay-what-you-use storage service that can support the high throughput demands of highly parallel applications.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {216007,
author = {Ana Klimovic and Yawen Wang and Christos Kozyrakis and Patrick Stuedi and Jonas Pfefferle and Animesh Trivedi},
title = {Understanding Ephemeral Storage for Serverless Analytics},
booktitle = {2018 USENIX Annual Technical Conference (USENIX ATC 18)},
year = {2018},
isbn = {978-1-939133-01-4},
address = {Boston, MA},
pages = {789--794},
url = {https://www.usenix.org/conference/atc18/presentation/klimovic-serverless},
publisher = {USENIX Association},
month = jul
}

Presentation Audio