GRPO用户终于有了解决训练停滞的实用工具——AVSPO无需额外推理成本就能提升4-6个点准确率，做大模型RL训练的团队可以直接试。
Group Relative Policy Optimization (GRPO) 在提升大语言模型推理能力方面表现出色，但存在优势坍塌问题：当组内奖励同质化（如全对或全错）时，优势趋近于零，导致梯度消失。研究者首次提出诊断指标 Advantage Collapse Rate (ACR)，量化训练批次中梯度无效的比例，并在0.5B至14B参数模型上验证了ACR对训练停滞和最终性能的强预测性。为缓解该问题，他们提出 Adaptive Virtual Sample Policy Optimization (AVSPO)，通过实时ACR监控注入虚拟奖励样本，无需额外模型推理即可从同质组中学习。AVSPO将优势坍塌减少58-63%，在所有模型规模上带来4-6个百分点的准确率提升，且保持了域外泛化能力。代码和数据集已开源。
Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs). However, GRPO is prone to advantage collapse, a failure mode where homogeneous rewards within a group (e.g., all correct or all incorrect answers) yield near-zero advantages and vanishing gradients. To address this, we introduce the Advantage Collapse Rate (ACR), the first diagnostic metric quantifying the proportion of training batches with ineffective gradients. Across models from 0.5B to 14B parameters on mathematical reasoning benchmarks, we show that ACR strongly predicts training stagnation and final performance. We then propose Adaptive Virtual Sample Policy Optimization (AVSPO), a lightweight extension of GRPO that injects virtual reward samples, guided by real-time ACR monitoring, to enable learning from homogeneous groups without additional model rollouts. AVSPO reduces advantage collapse by 58-63% relative to GRPO and yields consistent accuracy gains of 4-6 percentage points across all model scales, while maintaining generalization on the evaluated out-of-domain task. Code and datasets are available at https://qingyonghu.github.io/AVSPO.