ECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native Sparse Attention LLMs

Guangda Liu, Wenhao Chen, Chengwei Li, and Zhenyu Ning, Shanghai Jiao Tong University; Jing Lin and Yiwu Yao, Huawei; Quan Chen, Shixuan Sun, Jieru Zhao, and Minyi Guo, Shanghai Jiao Tong University