Towards Plan-aware Resource Allocation in Serverless Query Processing

Authors: 

Malay Bag, Alekh Jindal, and Hiren Patel, Microsoft

Abstract: 

Resource allocation for serverless query processing is a challenge. Unfortunately, prior approaches have treated queries as black boxes, thereby missing significant resource optimization opportunities. In this paper, we proposed a plan-aware resource allocation approach where the resources are adaptively allocated based on the runtime characteristics of the query plan. We show the savings opportunity from such an allocation scheme over production SCOPE workloads at Microsoft. We present our current implementation of a greedy version that periodically estimates the peak resource for the remaining of the query as the query execution progresses. Our experimental evaluation shows that such an implementation could already save more than 8% resource usage over one of our production virtual clusters. We conclude by opening the discussion on various strategies for plan-aware resource allocation and their implications on the cloud computing stack.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {254144,
author = {Malay Bag and Alekh Jindal and Hiren Patel},
title = {Towards Plan-aware Resource Allocation in Serverless Query Processing},
booktitle = {12th {USENIX} Workshop on Hot Topics in Cloud Computing (HotCloud 20)},
year = {2020},
url = {https://www.usenix.org/conference/hotcloud20/presentation/bag},
publisher = {{USENIX} Association},
month = jul,
}

Presentation Video