这篇论文讲了个很实在的优化：让空闲的解码节点分担预填充活儿，不用等KV传输，TTFT降了81%
该论文针对分离式LLM服务中预填充节点过载而解码节点空闲的问题，提出了一种主动预填充偏转调度器。在2个预填充节点和2个解码节点的A100集群上，预填充执行仅占P95首次令牌延迟的2-23%，其余为排队和KV-cache传输。该调度器让解码节点以分块预填充步骤穿插解码批次的方式处理请求，消除节点间KV传输。基于vLLM和DeepSeek-V2-Lite的实验显示，相比最优分离式调度器，P95 TTFT降低81%，SLO达成率提升79%。
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering. In practice, this creates a new asymmetry: under bursty, heavy-tailed workloads prefill nodes saturate while decode nodes have compute underutilized, and on a production-style A100 cluster with 2 prefill and 2 decode nodes (2P2D), we find that prefill execution accounts for only 2-23% of P95 Time-to-First-Token (TTFT). Queuing and inter-node GPU-GPU KV-cache transfer account for the rest. We present a proactive prefill-deflecting scheduler that lets decode nodes serve prefill phase of requests as chunked-prefill steps interleaved with their in-flight decode batches. For each queued request, we estimate the TTFT it would see on the prefill node, and on every decode node, search for the largest chunk schedule that keeps in-flight decodes within their Time-Between-Tokens (TBT) SLO and deflect when the decode path helps tail latency. Because the prefill phase of deflected requests runs in place on the decode node, the inter-node KV transfer is eliminated. Implemented on vLLM and evaluated on production-style traces with DeepSeek-V2-Lite, our approach reduces P95 TTFT by upto 81% and raises SLO attainment by upto 79% over state-of-the-art disaggregated schedulers, at sub-millisecond per-request routing cost.