PapersSelected

VelServe 用尾延迟校准调度解决 LLM 解码抖动问题

VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

Why we picked it

做推理服务部署的工程师应该看看这篇。它不换模型,只改调度:按实测尾延迟给每个解码步留余量,Qwen3-8B 上 SLO goodput 直接翻到基线的 2.4–2.9 倍,8 卡分离部署下达标请求数翻倍。比 vLLM 这类常规 SLO 调度器稳,适合流式和实时场景。

arXiv 论文提出 VelServe,一个针对交互式 LLM 服务 SLO(TTFT 和 TBT)的调度器,解决解码阶段的抖动问题。

Body

核心机制是根据每次迭代延迟的实测尾部分布,校准每个解码步可承载的 prefill 工作量,按 SLO 要求的迟到概率预留余量,防止一步超时拖累整个 batch。由于预留余量会拖慢 prefill,VelServe 优先服务等待时间最短的 prompt;

在 prefill 与解码分离部署下,它通过控制请求准入间隔,保证没有请求等待超过其 P99 目标允许的慢准入步数。实验覆盖三种注意力架构和多种负载:在 Qwen3-8B 和 DeepSeek-V4-Flash 上,VelServe 达到最强基线 2.

4–2.9 倍的 SLO goodput;

在 DeepSeek-V2-Lite 上至少 4.4 倍。

在 8 张 GPU 上做 prefill/decode 分离时,其满足 SLO 的请求数超过现有最先进基线的 2 倍。

Original · arXiv: DeepSeek

VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

Interactive LLM services must meet latency service-level objectives (SLOs) for the time to first token (TTFT) and the time between tokens (TBT), as streaming and soft real-time applications require predictable token delivery. We identify decode jitters in both aggregated and disaggregated serving. A jitter delays every request in the batch, and long responses span hundreds of steps, so even rare overruns cause many SLO violations. In aggregated serving, SLO-aware schedulers set each step's prefill budget until its predicted latency just meets the TBT target, leaving no room for mispredictions or host stalls. Disaggregation meets average TBT targets at a cost in capacity, yet the steps that admit new requests still run late. We present \sysname, a scheduler that calibrates how much prefill work each decode step carries using the measured tail of its own iteration latency. The SLO determines how rare a late step must be, and \sysname leaves enough headroom in each iteration to prevent overruns and meet the target decode latency. This headroom slows prefill, so \sysname serves the shortest waiting prompt first. When TTFT deadlines grow with prompt length, the long prompts it delays have the most time to spare. Under disaggregation, it spaces admissions so that no request waits on more slow admission steps than its P99 target allows. On three attention families and varying workloads, \sysname sustains 2.4--2.9$\times$ the SLO goodput of the strongest baseline on Qwen3-8B and DeepSeek-V4-Flash, and at least 4.4$\times$ on DeepSeek-V2-Lite. With prefill and decode disaggregated on eight GPUs, \sysname meets the SLO for more than twice as many requests as state-of-the-art baselines.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include VelServe, Qwen3-8B, DeepSeek, LLM inference.

Primary source verification: published directly by arXiv: DeepSeek with first-party authority. Cross-source corroboration: 2 independent sources have covered this development (including DAIR.AI), confirming industry-wide signal strength.