做推理加速的团队终于有了一个不牺牲接受率的剪枝方案——Graft用检索补偿剪枝损失，直接提升EAGLE-3 21.8%的加速比，搞LLM部署的值得试试。
投机解码通过草稿-验证范式加速大模型推理，但现有方法构建的庞大草稿树带来严重的显存带宽和计算开销。动态深度剪枝虽能减少延迟，却会丢弃潜在有效候选，导致接受率无法达到稠密树的上限。本文提出Graft框架，将剪枝与检索作为相互增强的操作：剪枝释放计算预算，检索补偿剪枝导致的覆盖损失并恢复接受长度。Graft采用“先剪后接”机制，以近乎零开销将高预测性的检索令牌填入剪枝空位，无需训练且无损。在短上下文、长上下文及大规模模型（如Qwen3-235B）上，Graft实现了最高5.41倍加速，平均加速比EAGLE-3提升21.8%，并初步探索了在非自回归草稿范式中的应用。
Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding
Speculative decoding (SD) accelerates large language model inference by leveraging a draft-then-verify paradigm. To maximize the acceptance rate, recent methods construct expansive draft trees, which unfortunately incur severe VRAM bandwidth and computational overheads that bottleneck end-to-end speedups. While dynamic-depth pruning can reduce this latency by removing marginal branches, it also discards potentially valid candidates, preventing the acceptance rate from reaching the upper bound of dense trees. In this paper, we identify a critical opportunity in resource allocation: the transition from dense to pruned drafting frees up significant computational budget. To break this Pareto tradeoff, we introduce Graft, a compensation framework that couples pruning and retrieval as mutually reinforcing operations. Pruning supplies sufficient budget for retrieval, while retrieval compensates for pruning-induced coverage loss and recovers accepted length. By employing a sequential `prune-then-graft' mechanism, Graft attaches highly predictive retrieved tokens into positions opened by pruning, filling the topological gaps with near-zero overhead. Graft is entirely training-free and lossless. Comprehensive evaluations show that Graft establishes a new Pareto frontier across practical deployment settings, including short-context generation, long-context generation, and large-scale models. On short-context benchmarks, it achieves up to 5.41$\times$ speedup and improves average speedup over EAGLE-3 by up to 21.8% on the large-scale Qwen3-235B. We also provide a preliminary exploration of applying Graft to the DFlash-style block drafting paradigm, offering initial evidence and insights for extending grafting beyond autoregressive draft trees.