Callo:在 prefill-decode 分离架构下利用失效 KV 加速缓存未命中请求
Callo: Serving Cache-Miss Requests with Speculative Decoding on Invalidated KV under Prefill-Decode Disaggregation
跑智能体服务的朋友应该看看这篇。上下文一改缓存就作废,vLLM 只能干等重算,Callo 拿废弃 KV 先投机生成再验证,实测对 vLLM 提速 1.05-1.19 倍。做推理服务优化的工程师和智能体平台团队都能直接参考。
学术论文 Callo 针对智能体会话中频繁的上下文编辑问题提出改进方案。上下文编辑会使已缓存的 KV 状态失效,请求必须等待重新 prefill 才能继续解码,带来额外延迟。
Callo 的核心思路是把失效的 KV 直接用于投机解码的 draft 阶段,等重新 prefill 完成后再用精确 KV 对草稿 token 逐一验证,通过才提交。系统还会在解码资源空闲时段调度投机计算,并在预判草稿接受率低时主动抢占,避免浪费算力。
评估在双 GPU 测试床上进行,使用 SWE-agent 会话和三个模型家族,对比 vLLM 在中等负载下取得 1.05-1.19x 的全请求加速;基于真实 trace 的集群级仿真显示该收益可随规模扩大保持。
Callo: Serving Cache-Miss Requests with Speculative Decoding on Invalidated KV under Prefill-Decode Disaggregation
Agent sessions may involve frequent context edits, which effectively invalidate cached KV state and force requests to wait for re-prefill before decoding. We present Callo, a disaggregated serving system that leverages invalidated KV for speculative drafts to reduce this latency, while verifying drafts against the exact KV after prefill before committing tokens. Callo makes effective use of decode resources by scheduling speculations during periods of underutilization and preempting speculations when drafts are likely to not be accepted. We evaluated Callo on a two-GPU testbed using SWE-agent sessions across three model families, observing all-request speedups of 1.05-1.19x over vLLM at moderate load. We demonstrate that these gains are also realized at scale through cluster-level simulations driven by real-world traces.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include speculative decoding, vLLM, KV Cache, inference services.
Source attribution: curated from arXiv cs.LG (@Jianjun Ou, Matthew Lentz).