这篇论文的SPIRAL方法教模型自己学会并行思考再汇总，比单纯加大顺序推理高效11倍，效果还更好，值得做推理扩展的朋友看看。
SPIRAL提出一种新训练框架，让语言模型在推理时同时使用顺序链式思维、平行采样和最终聚合三种原语。该方法通过集束强化学习优化所有组件，在推理任务中扩展效果优于GRPO，最高实现11倍扩展效率和15%性能提升。实验表明模型能有效学习生成对聚合有用的轨迹集并改进最终答案。
SPIRAL: Learning to Search and Aggregate
Language model reasoning can be substantially improved at test time via scaffolds that scale inference compute across different primitives -- sequential reasoning within a trace, independently sampled parallel traces, and aggregation of multiple reasoning traces into a final response. During post-training, however, language models are optimized only for sequential reasoning within a single trace. We introduce Sequential-Parallel-Aggregative Reinforcement Learning (SPIRAL), a framework in which a language model is trained to use all three primitives, as part of a unified inference compute pipeline. Concretely, the language model first samples a set of independent traces in parallel, each produced through sequential chain-of-thought reasoning, and then generates a final aggregation trace conditioned on those traces; all components are optimized end-to-end against the reward of the final aggregated response. To train this system, SPIRAL uses set reinforcement learning to teach models to produce a set of traces that are collectively useful for an aggregator and standard reinforcement learning to teach models to aggregate the set into improved final responses. Our experiments on reasoning tasks show that SPIRAL effectively scales with inference compute, outperforming GRPO by up to 11$\times$ scaling efficiency and 15% higher performance when all three compute primitives are scaled.