论文揭示前缀缓存的智能体服务存在亚稳态：一次突发可致吞吐腰斩70分钟
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
跑过 vLLM 智能体服务的人应该都见过那种莫名的性能崩塌，这篇论文把它解释清楚了：一次几小时的突发流量就能让命中率掉到 10%，吞吐腰斩还自己好不了。论文给了能用的预算器方案，不用调参，直接把 KV 池撑爆的可能性掐掉。
arXiv 论文研究了前缀缓存（prefix caching）在智能体推理服务中的失效模式：一次命中只需增量预填充约一千 token，一次未命中则要重算 10–40K token 的上下文。实验显示，在相同基础负载下，几分钟的额外到达就会让 vLLM 服务器进入退化状态——缓存命中率降至约 10%，吞吐量减半，积压在剩余 70 分钟内持续增长，而无突发的对照组保持健康（12 对中 9 对如此，反向从未出现）。
机制在于：服务变慢使更多会话滞留在系统中，工作集超出 KV 缓存池，之后每一轮都变成全量重算。该退化在缓存预热服务器本可承受的负载下出现，无前缀缓存时不存在，用 Qwen3-32B 跑 SWE-bench 智能体轨迹、换用 SGLang 引擎同样能复现，SGLang 需要更大的触发条件。
应对方面，限制新会话进入的限流策略只救回 7 个崩溃运行中的 1 个；按容量上限挂起已接入会话则全部在几分钟内恢复。CPU offload 层只能延迟而非阻止崩溃。作者提出的预算器将活跃会话的驻留 KV 需求限定在缓存池内，阈值无需按负载调参，在所有测试中都防止了故障。
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
LLM agents drive inference engines with closed-loop sessions whose context grows every turn, and serving them economically depends on prefix caching: a hit costs an incremental prefill of about a thousand tokens, a miss recomputes a 10--40K-token context. We show that this makes prefix-cached agent serving metastable. With an identical base workload, a few minutes of extra arrivals leaves a vLLM server in a degraded regime--a ~10% hit ratio, about half the throughput, and a backlog that grows for the remaining 70 minutes--while its no-burst twin stays healthy (9 of 12 pairs; none in the reverse direction). The regime appears at loads the cache-warm server sustains, is absent without prefix caching, and also appears with Qwen3-32B on SWE-bench agent trace shapes and on SGLang, which tolerates larger triggers. Slower service keeps more sessions in the system, their working set outgrows the KV pool, and every turn becomes a recomputation; a first-order workload-balance condition, calibrated independently, brackets when the regime can drain. Getting out is the harder problem: throttling new admissions recovered 1 of 7 matched collapsed runs, while suspending admitted sessions against a capacity-based limit recovered all of them within minutes. A CPU offload tier delays the collapse but does not prevent it. A budgeter that bounds the resident KV demand of active sessions to the pool, with thresholds that need no per-workload tuning, prevented the failure in every run we tested.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include vLLM, SGLang, prefix caching, KV缓存.
Source attribution: curated from arXiv cs.LG (@Wen Huang, Jack Liu).
