MCPO：按模型犹豫程度分配强化学习奖励的 token 级信用方法
Margin-Guided Token Credit for Reinforcement Learning with Verifiable Rewards
GRPO 把答案奖励平均撒给每个 token，这篇论文改成按模型犹豫程度（top-2 概率差）分配信用，不用额外评估。在 MiniCPM5-2B 和 Qwen3-4B 上平均涨 6.55 分，pass@32 涨 13.8。做 RLVR 训练、想榨干小模型数学能力的人可以直接抄这套实现。
一篇 arXiv 论文提出 MCPO（Margin-Credit Policy Optimization），改进 GRPO 在可验证奖励强化学习中把最终答案奖励平均分给每个 token 的做法。方法用策略下一个 token 分布的 top-2 边际 M=p1-p2 衡量模型犹豫程度，经两个固定阈值映射为分数，把每条解法的 advantage 按犹豫程度分配到各 token，同时保持均值不变。
该边际分数直接从训练器已有的前向传播读取，无需额外模型评估，也不依赖批次统计量。作者用 LoRA 在 2,048 道 DAPO-Math 问题上训练 MiniCPM5-2B-Base 共 200 步，在六个竞赛数学基准上评测：MCPO 在全部五个数据种子上超过未修改的 GRPO 基线，macro avg@32 平均提升 6.55 分（95% CI [4.1, 9.0]）；Qwen3-4B-Base 复现实验在三个种子上均提升 4.75 分。
收益在 GRPO 只能部分解出的问题上最大，且随 k 增长，pass@32 时达 13.8 分。MCPO 训练的模型答题长度是基线的两到三倍，写得更长的运行得分更高，但论文未将这两个效应分离。
Margin-Guided Token Credit for Reinforcement Learning with Verifiable Rewards
In reinforcement learning with verifiable rewards, a language model is rewarded for a correct final answer, and GRPO passes that reward to every token of the solution in equal measure. We instead place the credit where the model hesitated, in a method we call MCPO (Margin-Credit Policy Optimization). We measure hesitation by the top-2 margin $M=p_1-p_2$ of the policy's next-token distribution, map it to a score with two fixed thresholds, and spread each solution's advantage over its tokens so that the mean stays unchanged. The margin is read from a forward pass the trainer already performs, so the method needs no extra model evaluation and no batch statistics. We train MiniCPM5-2B-Base with LoRA for 200 steps on 2,048 DAPO-Math problems and evaluate on six competition-math benchmarks. MCPO beats an unmodified GRPO baseline for all five data seeds, by 6.55 points of macro avg@32 on average (95\% CI $[4.1, 9.0]$), and a Qwen3-4B-Base replication improves for all three seeds ($+4.75$). The gain is largest on problems that GRPO solves only some of the time and grows with $k$ in pass@$k$, reaching 13.8 points at pass@32. MCPO models also write two to three times longer answers, and runs that write longer score higher; our experiments do not separate the two effects.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include MCPO, GRPO, Qwen3, 强化学习.
Source attribution: curated from arXiv cs.LG (@Chi Liu, Xin Chen, Yuchuan Weng, Srinivasan Manoharan).