LoopFill:面向 Looped Transformer 的免训练流水线多 token 推测解码
LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers
搞推理优化的可以看看这篇:LoopFill 把两种草稿器合到一棵树里,免训练就在 Ouro-2.6B 上跑到 5.85 倍加速,比 LoopAhead 快 25%-40%。做 Looped 模型部署或推测解码研究的都该读读原文。
论文提出 LoopFill,一种针对 Looped LLM 的免训练推测解码算法。在 Ouro-2.6B 上相对贪心自回归解码平均加速 5.85 倍,在 Huginn-3.5B 上加速 5.13 倍,比最强基线 LoopAhead 分别高出 40% 和 25%,结果覆盖七个基准。
技术核心是把多 token 后缀草稿与浅层循环预测合并进同一棵草稿树:后缀查找可以直接扩展浅层预测,多个未来 token 一起进入流水线,共享块在一个 batch 里对不同循环深度的候选打分,再用全深度验证保持输出分布不变,且无需额外训练。
这解决了 Looped Transformer 每生成一个 token 需多次串行过层导致的延迟问题,也突破了以往流水线方法中 token 必须达到草稿深度才能进入下一级的限制,对依赖参数高效模型的部署方有直接加速价值。
LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers
Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. During autoregressive decoding, however, generating each token requires several sequential passes through these layers, leading to high decoding latency. Pipelined self-speculative methods use shallow-loop predictions to process tokens at different loops in the same batch, but each token must reach the drafting depth before the next can enter the pipeline. We introduce LoopFill, a training-free speculative decoding algorithm for looped LLM that combines multi-token suffix drafts with shallow-loop predictions in a single draft tree. Both drafters contribute to one tree, so a suffix lookup can immediately extend a shallow prediction and several future tokens can enter the pipeline together. The shared block computes the highest-scoring draft candidates at different loop depths in one batch, and full-depth verification preserves the model's autoregressive output distribution. Across seven benchmarks, LoopFill's average speedup over greedy autoregressive decoding is $5.85\times$ on Ouro-2.6B and $5.13\times$ on Huginn-3.5B. The speedups exceed those of LoopAhead, the strongest baseline, by \(40\%\) on Ouro-2.6B and \(25\%\) on Huginn-3.5B.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LoopFill, Looped Transformer, speculative decoding, Ouro.
Source attribution: curated from arXiv cs.AI (@Siqi Zhu, Suozhi Huang, Yi Zhao, Jiaxuan You).