Stint:让长输出 LLM 服务中的 KV 缓存抢占不再浪费算力
Making Preemption Productive in Long-Output LLM Serving
跑推理服务的人都知道内存被打满后请求互相挤兑有多难受。Stint 把被抢占请求的 KV 快照挪到 CPU 再搬回来,卡一分钟以上的请求直接消失,p99 延迟砍半。自己搭 vLLM/SGLang 之类服务的人值得看看思路。
Stint 是一个针对推理型 LLM 长输出场景的服务系统,解决 KV 缓存占满 GPU 内存后请求被反复抢占、状态丢失又恢复的 KV thrashing 问题。论文指出,现有系统在某些情况下会让超过五分之一的请求卡住超过一分钟。
技术机制上,Stint 将被抢占请求的 KV 缓存以定长快照形式存入 contiguous-first 的 CPU 内存池,通过 exact-extent 分配器管理,配合融合的 K/V gather/scatter 内核与双缓冲搬运数据;读回时按年龄排序,并给恢复的请求一轮保护性解码,避免立刻再次被抢占,同时用解码隐藏状态估计剩余长度来选择牺牲请求。
实测中,在 H200 GPU 上用 Qwen3-32B 回放奥数数学题场景,Stint 下没有请求停顿超过 50 秒,p99 每输出 token 延迟相对同引擎的重计算方案下降 50.7%,整体回放提前 10.7% 完成,吞吐提升 12.0%。
Making Preemption Productive in Long-Output LLM Serving
Reasoning LLMs generate long outputs whose key--value (KV) caches occupy GPU memory for minutes. When memory fills, the server preempts requests, and a preempted request can repeatedly lose and recover its state while producing little new output. We call this behavior KV thrashing. In certain cases, existing systems leave more than a fifth of requests stalled for over a minute. Because preemption cannot be avoided without delaying admission, we ask how to make it productive. We present Stint, a serving system that preserves a preempted request's work, bounds how long it waits, and gives it fresh output after it returns. Stint stores fixed-length KV snapshots in a contiguity-first CPU pool with an exact-extent allocator, and moves them with fused K/V gather/scatter kernels and double buffering. Age-based readmission brings stalled requests back, and protection gives them a turn of fresh decoding before another preemption. Remaining-length estimates from decoding hidden states guide victim selection. On olympiad-math replays with Qwen3-32B on H200 GPUs, no request pauses for more than 50 seconds under Stint and p99 time per output token falls by 50.7\% relative to recomputation in the same engine. The replay as a whole finishes 10.7\% sooner (12.0\% higher throughput), because the requests that baselines starve no longer finish last.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Stint, Qwen3-32B, KV cache, inference services.
Source attribution: curated from arXiv cs.AI (@Ziji Shi, Xinyi Wan, Chaoyi Ruan, Guangxing Huang, Min Lin, Jialin Li). Cross-source corroboration: 2 independent sources have covered this development (including Gorden Sun), confirming industry-wide signal strength.
- Gorden Sun11-15 03:26Source