XM-Sparse Boosts Sparse-Attention LLM Serving via Scale-up Memory
XM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory Pools
The finding that K-index caches, not just KV caches, cap decode throughput reframes how sparse-attention serving systems should be architected, and the dual I/O strategy offers a concrete blueprint. Engineers running SGLang with long-context models like DeepSeek-V3.2 or GLM-5.3 can apply these techniques to raise concurrency without adding GPUs. Hardware teams evaluating X-Mem-style memory expansion get a quantified baseline against PCIe offloading alternatives.
A new paper introduces XM-Sparse, the first sparse-attention KV offloading system built on scale-up memory pools such as X-Mem, addressing a key bottleneck in long-context LLM serving. Experiments with DeepSeek-V3.2 and GLM-5.3 on an emulated scale-up memory pool show decode throughput gains of up to 4.98x over PCIe-based ECHO, 4.52x over HiSparse, and 1.34x over a naive integration on the same platform, across context lengths from 16K to 1M tokens.
The authors first analyse why offloading alone is insufficient: even with scale-up memory pools, decode throughput stays bounded by GPU HBM capacity because hot KV buffers and the fully resident indexer key (K-index) cache occupy substantial HBM. XM-Sparse offloads both caches and matches each with a tailored I/O mechanism. Dynamically selected KV entries are retrieved via TMA fused with attention in a tile-level pipeline to hide on-demand latency, while predictable K-index accesses are prefetched through SM-efficient GPU-initiated DMA, coordinated with latency-critical operations to avoid disrupting decoding.
The system is implemented atop SGLang, an open-source serving framework widely used in production. Sparse attention is increasingly adopted by production LLMs for long-context inference, but serving systems still retain full KV history, making HBM capacity the limit on concurrency; scale-up memory pools relieve both PCIe bandwidth and host memory capacity constraints.
XM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory Pools
Sparse attention has been increasingly adopted by production LLMs for long-context inference, yet serving systems must still retain the full KV history, making GPU HBM capacity a key limit on serving concurrency. Existing systems offload the KV cache to host memory, but suffer from the limited bandwidth of PCIe and the limited capacity of host memory. Emerging scale-up memory pools (e.g., X-Mem) relieve both limits and open new opportunities for sparse-attention offloading. In this paper, we first conduct an in-depth analysis and reveal that, even with scale-up memory pools, decode throughput remains bounded by HBM capacity, as hot KV buffers and the fully resident indexer key (K-index) cache occupy substantial HBM. Moreover, the two caches exhibit distinct access patterns. Based on these insights, we present XM-Sparse, the first sparse-attention offloading system over scale-up memory expansion. The core idea of XM-Sparse is to offload both caches aggressively while matching each with an I/O mechanism tailored to its access pattern. Specifically, for dynamically selected KV entries, XM-Sparse fuses TMA-based retrieval with attention in a tile-level pipeline to hide on-demand retrieval latency. For predictable K-index accesses, it prefetches pages via SM-efficient GPU-initiated DMA and coordinates prefetching with latency-critical operations to avoid interfering with decoding. We have implemented XM-Sparse atop SGLang, and our experiments on DeepSeek-V3.2 and GLM-5.3 with an emulated scale-up memory pool show that XM-Sparse delivers up to 4.98$\times$ and 4.52$\times$ higher decode throughput than PCIe-based ECHO and HiSparse, respectively, and up to 1.34$\times$ over a naive integration on the same platform, across 16K-1M contexts.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include DeepSeek-V3.2, GLM-5.3, SGLang, sparse attention.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority.