In support of workload-aware streaming state management

Authors: 

Vasiliki Kalavri, Boston University; John Liagouris, Boston University & Hariri Institute for Computing
Outstanding New Research Direction Award Winner!
Best Presentation Award Finalist

Abstract: 

Modern distributed stream processors predominantly rely on LSM-based key-value stores to manage the state of long-running computations. We question the suitability of such general-purpose stores for streaming workloads and argue that they incur unnecessary overheads in exchange for state management capabilities. Since streaming operators are instantiated once and are long-running, state types, sizes, and access patterns, can either be inferred at compile time or learned during execution. This paper surfaces the limitations of established practices for streaming state management and advocates for configurable streaming backends, tailored to the state requirements of each operator. Using workload-aware state management, we achieve an order of magnitude improvement in p99 latency and 2x higher throughput.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {254282,
author = {Vasiliki Kalavri and John Liagouris},
title = {In support of workload-aware streaming state management},
booktitle = {12th {USENIX} Workshop on Hot Topics in Storage and File Systems (HotStorage 20)},
year = {2020},
url = {https://www.usenix.org/conference/hotstorage20/presentation/kalavri},
publisher = {{USENIX} Association},
month = jul,
}

Presentation Video