长上下文推理，内存传输是瓶颈。新方案SAC用CXL按需取KV缓存，比RDMA吞吐量翻倍、延迟降到十分之一，做稀疏推理的值得一看。
长上下文LLM推理的内存瓶颈日益突出。传统RDMA解耦内存池对于稀疏注意力模型效率低下，仍需完整获取KV缓存。SAC系统利用CXL的低延迟、缓存行粒度加载/存储语义，仅在推理时按需获取所需的top-k KV条目。在DeepSeek-V3.2上使用SGLang的评估显示，相比RDMA基线，SAC实现了2.1倍吞吐量提升、9.7倍TTFT降低和1.8倍TBT降低。
SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL
The scaling of LLMs toward long-context inference has shifted the primary serving system bottleneck from computation to memory capacity. Traditional solutions for dense attention models rely on RDMA-based disaggregated memory pools, which perform coarse-grained fetching of the entire prefix KV cache from remote storage to local memory before decoding. However, this approach is fundamentally inefficient for emerging sparse attention models. While only a small fraction of KV entries are active during decoding, these systems still fetch the full KV cache locally, leading to severe transmission bottlenecks and local memory wastage. To address this, we propose SAC, the first efficient disaggregated KV cache system optimized for sparse attention models. By leveraging the low-latency, cache-line granularity load/store semantics of Compute Express Link (CXL), SAC fetches only the required top-k KV entries on demand during inference. Evaluations on DeepSeek-V3.2 using SGLang show that SAC achieves 2.1x higher throughput, 9.7x lower TTFT, and 1.8x lower TBT compared to RDMA-based baselines, establishing CXL-based disaggregation as the superior infrastructure for emerging sparse attention models.