Auto-sizing for Stream Processing Applications at LinkedIn

Authors: 

Rayman Preet Singh, Bharath Kumarasubramanian, Prateek Maheshwari, and Samarth Shetty, LinkedIn Corp

Abstract: 

Stream processing as a platform-as-a-service (PaaS) offering is used at LinkedIn to host thousands of business-critical applications. This requires service owners to manage applications' resource sizing and tuning. Unfortunately, applications have diverged from their conventional model of a directed acyclic graph (DAG) of operators and incorporate multiple other functionalities, which presents numerous challenges for sizing. We present a controller that dynamically controls applications' resource sizing while accounting for diverse functionalities, load variations, and service dependencies, to maximize cluster utilization and minimize cost. We discuss the challenges and opportunities in designing such a controller.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {254150,
author = {Rayman Preet Singh and Bharath Kumarasubramanian and Prateek Maheshwari and Samarth Shetty},
title = {Auto-sizing for Stream Processing Applications at LinkedIn},
booktitle = {12th {USENIX} Workshop on Hot Topics in Cloud Computing (HotCloud 20)},
year = {2020},
url = {https://www.usenix.org/conference/hotcloud20/presentation/singh},
publisher = {{USENIX} Association},
month = jul,
}

Presentation Video