TideCache:面向混合架构 LLM 的 CPU-GPU 缓存调度系统
TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming
SGLang 团队新出的缓存调度方案,专门解决混合架构模型缓存难管的问题。实测 agent 吞吐最高翻倍,TTFT 提升三倍。自己搭服务跑 agent 或长上下文的开发者直接能抄作业。
学术论文 TideCache 针对 open-weight 模型中混合架构(global-attention 与 bounded-state 层)缓存需求异构的问题,提出一套在 CPU-GPU 超级芯片上的 LLM 服务系统。系统包含三个组件:Transparent Cache Pool 通过固定虚拟地址在缓存组件间重分配物理页;Cache Streamer 利用 bounded-state 层的计算间隙预取下一层 global-attention 的 prefix,并把逐层传输与计算重叠;Request Scheduler 根据请求大小和队列状态选择常驻或流式执行。这套机制让流式释放的 HBM 容纳更多缓存组件和请求。
TideCache 实现在 SGLang 上,并在 GB300 和 GH200 上跨七种模型/平台配置完成评估。与 SGLang + HiCache 相比,TideCache 完成的 agent 轮次提升至最高 2.04 倍,整体 P50 per-token 延迟降低 20–59%,agent P50 TTFT 提升 1.40–3.05 倍。对部署智能体长上下文工作负载的团队,这意味着同样硬件下更高的并发与更低的延迟。
TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming
Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU--GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory. The Transparent Cache Pool redistributes physical pages among cache components behind fixed virtual addresses, preserving cache layouts and backend interfaces. The Cache Streamer uses computation intervals provided by bounded-state layers to prefetch the next global-attention layer's prefix into GPU slots shared across global-attention layers, reducing GPU residency while overlapping layer-wise transfers with computation. Generated KV remains resident. The Request Scheduler selects resident or streamed execution and triggers capacity adjustments according to request size and queue state. Together, they let HBM released by streaming accommodate other cache components and additional requests. Implemented in SGLang and evaluated on GB300 and GH200 across seven model/platform configurations, TideCache completes up to 2.04x as many agent turns, reduces overall P50 per-token latency by 20--59%, and improves agent P50 TTFT by 1.40--3.05x compared with SGLang with HiCache.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include TideCache, SGLang, KV cache, reasoning models.
Source attribution: curated from arXiv cs.LG (@Jinbin Luo, Xiaozhe Yao, Shang Yang, Zhijian Liu, Song Han, Qinghao Hu).