这篇论文把 GRPO 和 Dr. GRPO 的底裤都扒了——原来两者各占一个极端，没法同时做到无偏和长度无关，搞推理模型训练的人得看看这个权衡。
GRPO 被 DeepSeek-R1 采用作为训练推理能力的主要强化学习算法，但存在响应级长度偏差。最新改进 Dr. GRPO（COLM 2025）声称通过移除长度归一化实现“无偏”，但作者证明该声称不完整。文章提出不可能定理：在标准结果奖励加 GRPO 设定下，没有长度加权方案能同时实现梯度无偏估计（P1）和长度不变性（P2）。通过参数族 f_α(L)=L^{α-1} 显示，α=0 对应 GRPO（近似满足 P2 但违反 P1），α=1 对应 Dr. GRPO（满足 P1 但违反 P2），且 Dr. GRPO 的长度偏差导致较长轨迹的梯度贡献与长度比成正比。结果表明两种算法处于不可避免的权衡两端。
On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
Group Relative Policy Optimization (GRPO) is the dominant reinforcement learning algorithm for training reasoning capabilities in large language models, notably adopted by DeepSeek-R1. The recent improvement Dr. GRPO (COLM 2025) identifies the response-level length bias caused by per-trajectory length normalization in GRPO and proposes removing this normalization, claiming the resulting optimizer is "unbiased." We show that this claim is incomplete. Specifically, we establish an impossibility theorem: under the standard outcome reward + GRPO setting, no length-based weighting scheme can simultaneously achieve the following two properties. (P1) Gradient unbiasedness: the gradient estimator is an unbiased estimate of the true policy gradient. (P2) Length invariance: each trajectory's effective contribution to the gradient is independent of its token length. GRPO approximately satisfies P2 but violates P1; Dr. GRPO satisfies P1 but violates P2. We characterize the complete tradeoff spectrum via the parametric family f_alpha(L) = L^{alpha - 1}, where alpha = 0 recovers GRPO, alpha = 1 recovers Dr. GRPO, and provide quantitative analysis showing that Dr. GRPO's length bias can cause longer trajectories to dominate gradient updates by a factor proportional to the length ratio. Our results reveal that neither algorithm is universally "done right"; they occupy opposite ends of a fundamental and unavoidable tradeoff.