Shan Yu, University of California, Los Angeles; Yifan Qiao, University of California, Berkeley; Mingyuan Ma, Harvard University; Yangmin Li, Carnegie Mellon University; Shuo Yang, University of California, Berkeley; Xinyuan Tong, University of Edinburgh; Yang Wang, Intel; Zhiqiang Xie, Stanford University; Yuwei An, Carnegie Mellon University; Shiyi Cao, University of California, Berkeley; Ke Bao, LMSYS; Deepak Vij, Xiaoning Ding, and Yichen Wang, ByteDance; Qingda Lu, Alibaba Cloud; Zhong Wang, Tsinghua University; Gao Gao, Novita AI; Harry Xu and Junyi Shu, University of California, Los Angeles; Jiarong Xing, Rice University; Ying Sheng, University of California, Los Angeles
Inference providers must maintain availability for many LLMs, including low-volume but essential models, making resource efficiency increasingly important as token prices fall. Analysis of production traces reveals a dynamic bursty-group pattern in which sets of models become active together and shift over time; existing space- and time-sharing approaches lack principled mechanisms to adapt to this variability, forcing trade-offs between SLO adherence and efficiency. We observe that elastic memory allocation can unify spatial and temporal sharing. Based on this insight, we have developed Prism, a memory-centric LLM co-serving framework that applies memory ballooning to reclaim memory across models and support both forms of sharing under a single scheme. Prism’s balloon driver, referred to as kvcached, has been open-sourced at https://github.com/ovg-project/kvcached, and deployed in production environments across 10K+ GPUs.
OSDI '26 Open Access Sponsored by
King Abdullah University of Science and Technology (KAUST)
Open Access Media
USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.
Yu PDF