做推理模型RL微调的团队终于有了降低训练方差的实际方案——RREDCoT用模型自身做信用分配，省去额外生成成本，长上下文场景下效果显著，值得关注。
当前推理语言模型通过强化学习微调时，常使用GRPO算法，但该算法面临延迟奖励问题——只能在完整思维链后给出奖励，导致高方差。RREDCoT提出一种无需额外生成的奖励再分配方法，利用模型自身对思维链中的关键片段进行信用分配，从而降低训练方差。实验表明，该方法在计算开销和效果上优于蒙特卡洛采样及其他归因方法。这项研究为提升推理模型训练效率提供了新思路，尤其适合长上下文场景。
RREDCoT: Segment-Level Reward Redistribution for Reasoning Models
Recent advancements in reasoning language models have been driven by Reinforcement Learning (RL) fine-tuning. Most often, these rely on the Group Relative Policy Optimization (GRPO) algorithm or modifications thereof to steer the models to produce Chain-of-Thought (CoT) traces. The final answer can only be verified, and the reward assigned, after the CoT trace is complete, making it a delayed reward problem. GRPO and its modifications correspond to Monte Carlo methods in standard RL, which are known to suffer from high variance. A possible solution to this problem is the redistribution of rewards through credit assignment, where segments of the CoT trace that are important for arriving at the desirable solution are emphasized by assigning a higher reward. While Monte Carlo sampling can be used to provide an unbiased estimate of intermediate state values, its computational overhead makes it unsuitable for train-time credit assignment in long contexts at high granularity. We introduce RREDCoT (Reward REDistribution for Chain of Thoughts), which utilizes the model itself to approximate the optimal reward redistribution without additional generation. We investigate the advantages of our method compared to MC sampling and several attribution methods. We further analyze several aspects relevant to the construction of the redistribution such as segmentation of CoT traces and state value estimation.