LeanSparse Cuts GPU Memory for Sparse-Attention LLM Serving
LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs
The key insight is that prefill memory spikes, not just KV cache size, throttle concurrency — streaming layer histories through reusable buffers removes that admission bottleneck. Engineers running long-context inference on SGLang can adopt the design to serve more users per GPU without changing model weights. Compared with HiSparse and ECHO, the gains come from better indexing rather than more hardware.
LeanSparse is a serving system for native sparse-attention models that reduces both decode-time persistent GPU memory and prefill-time temporary memory demand. During prefill it streams layer histories through two reusable GPU buffers, overlapping loading with model computation; during decode it uses coarse-to-fine indexing to retrieve original indexer keys only for candidate refinement. Implemented in SGLang and evaluated on DeepSeek-V4-Flash, it supports 2.95× more concurrent requests and 2.14× decode throughput over HiSparse at 950K context length. On multi-turn Open-SWE traces, it cuts median time to first token from 21.6s to 0.6s and median session completion time by 17.7% versus ECHO.
LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs
Sparse attention limits the history read by each query, but serving long contexts still requires substantial GPU memory. Existing KV offloading systems retain the full history of indexer keys on GPU, limiting decode concurrency. Prefill's temporary memory demand can further delay admission even when enough memory remains for the same request's subsequent decode. We present LeanSparse, a serving system that reduces both decode's persistent memory use and prefill's temporary memory demand. Our design exploits two observations: prefill consumes historical data one layer at a time, and decode can identify promising candidates before comparing their original indexer scores. During prefill, LeanSparse streams layer histories through two reusable GPU buffers, overlapping next-layer loading with ongoing model computation. During decode, coarse-to-fine indexing searches compact representations and retrieves original indexer keys only for candidate refinement. Separate indexer key and KV caches reuse data across generation steps. We implement LeanSparse in SGLang and evaluate it on DeepSeek-V4-Flash. At 950K context length, LeanSparse supports 2.95$\times$ as many concurrent requests and achieves 2.14$\times$ decode throughput over HiSparse. On multi-turn Open-SWE traces, it reduces median time to first token from 21.6 to 0.6s and median session completion time by 17.7% relative to ECHO.