RL 训练嫌生成太慢？SpecRoll 用投机解码给 rollout 提速，不改采样分布，五个模型跑数学推理端到端快 1.2 倍以上，代码也开源了。
SpecRoll 是一种用于强化学习 rollout 的投机解码引擎，面向大规模语言模型后训练。它用轻量级未来 token 头生成并行提议，并通过 Reflex 模块做轨迹局部的隐藏状态修正，无需反向传播。在 1.5B 到 14B 的五个模型和三个数学推理数据集上，SpecRoll 相比原始 GRPO 取得 1.26-2.15 倍生成加速和 1.21-2.04 倍端到端加速。在全部 15 组匹配设置中，SpecRoll 的平均端到端耗时比 FastGRPO 快 1.18 倍。源码已公开。
SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts
Reinforcement learning (RL) post-training improves the reasoning capabilities of large language models, but autoregressive rollout generation remains a major efficiency bottleneck. Speculative decoding can accelerate generation, yet applying it during RL is difficult because the target policy continually evolves: static proposers become stale, while frequent drafter updates add substantial overhead. We introduce SpecRoll, a speculative rollout engine that preserves the target model's sampling distribution while adapting at two timescales. Lightweight future-token heads generate parallel proposals, while our proposed Reflex module uses delayed verifier feedback to perform bounded, trajectory-local hidden-state corrections without backpropagation. A complementary slow path updates the head parameters only when sustained degradation is detected. SpecRoll combines these mechanisms with concurrency-aware sparse-tree verification and exact target verification, leaving the target rollout distribution and GRPO objective unchanged. Across five models ranging from 1.5B to 14B and three mathematical reasoning datasets, SpecRoll achieves 1.26-2.15x generation speedup and 1.21-2.04x end-to-end speedup over vanilla GRPO. It also outperforms FastGRPO in both generation and end-to-end time across all 15 matched settings, with an average pairwise end-to-end gain of 1.18x. Controlled ablations show that the fast and slow adaptation paths provide complementary benefits. Our source code is available at https://anonymous.4open.science/r/SpecRoll-26062006.