Papers

Stint:让长输出 LLM 服务中的 KV 缓存抢占不再浪费算力

Making Preemption Productive in Long-Output LLM Serving

Why we picked it

跑推理服务的人都知道内存被打满后请求互相挤兑有多难受。Stint 把被抢占请求的 KV 快照挪到 CPU 再搬回来,卡一分钟以上的请求直接消失,p99 延迟砍半。自己搭 vLLM/SGLang 之类服务的人值得看看思路。

Stint 是一个针对推理型 LLM 长输出场景的服务系统,解决 KV 缓存占满 GPU 内存后请求被反复抢占、状态丢失又恢复的 KV thrashing 问题。论文指出,现有系统在某些情况下会让超过五分之一的请求卡住超过一分钟。

Body

技术机制上,Stint 将被抢占请求的 KV 缓存以定长快照形式存入 contiguous-first 的 CPU 内存池,通过 exact-extent 分配器管理,配合融合的 K/V gather/scatter 内核与双缓冲搬运数据;读回时按年龄排序,并给恢复的请求一轮保护性解码,避免立刻再次被抢占,同时用解码隐藏状态估计剩余长度来选择牺牲请求。

实测中,在 H200 GPU 上用 Qwen3-32B 回放奥数数学题场景,Stint 下没有请求停顿超过 50 秒,p99 每输出 token 延迟相对同引擎的重计算方案下降 50.7%,整体回放提前 10.7% 完成,吞吐提升 12.0%。

Original · arXiv cs.AI

Making Preemption Productive in Long-Output LLM Serving

Reasoning LLMs generate long outputs whose key--value (KV) caches occupy GPU memory for minutes. When memory fills, the server preempts requests, and a preempted request can repeatedly lose and recover its state while producing little new output. We call this behavior KV thrashing. In certain cases, existing systems leave more than a fifth of requests stalled for over a minute. Because preemption cannot be avoided without delaying admission, we ask how to make it productive. We present Stint, a serving system that preserves a preempted request's work, bounds how long it waits, and gives it fresh output after it returns. Stint stores fixed-length KV snapshots in a contiguity-first CPU pool with an exact-extent allocator, and moves them with fused K/V gather/scatter kernels and double buffering. Age-based readmission brings stalled requests back, and protection gives them a turn of fresh decoding before another preemption. Remaining-length estimates from decoding hidden states guide victim selection. On olympiad-math replays with Qwen3-32B on H200 GPUs, no request pauses for more than 50 seconds under Stint and p99 time per output token falls by 50.7\% relative to recomputation in the same engine. The replay as a whole finishes 10.7\% sooner (12.0\% higher throughput), because the requests that baselines starve no longer finish last.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Stint, Qwen3-32B, KV cache, inference services.

Source attribution: curated from arXiv cs.AI (@Ziji Shi, Xinyi Wan, Chaoyi Ruan, Guangxing Huang, Min Lin, Jialin Li). Cross-source corroboration: 2 independent sources have covered this development (including Gorden Sun), confirming industry-wide signal strength.