LGPO：去掉重要性比率，也能稳定训练 LLM 强化学习
Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?
做 RL 后训练的值得看：LGPO 用似然门控替掉 GRPO 里重复算 old-policy 似然那步，训练时间少约 18%，全 on-policy 时 GRPO 会崩它还稳着。省显存又省算力，自己跑 RL 的团队可以直接对照论文改实现。
一篇新论文提出 Likelihood-Gated Policy Optimization（LGPO），用基于似然的门控机制实现软信任域约束，省去了 old-policy 似然的重复计算。GRPO 等方法需要同时计算 old-policy 和 current-policy 似然来构造重要性采样比率，带来不小开销。实验显示，去掉重要性采样修正项不损害训练稳定性，而去掉信任域机制会导致训练崩溃；比率裁剪在全 on-policy 训练中还会失效，因为重要性比率始终为 1，约束从未激活。在 GRPO 本身稳定的标准设定下，LGPO 达到相当的稳定性和峰值性能，训练时间平均减少约 18%。在全 on-policy 训练中 GRPO 会失效，LGPO 仍保持稳定。
这一结果指向一个重要判断：训练稳定性主要来自信任域约束而非重要性比率本身。对做 LLM 后训练（RL post-training）的团队来说，省掉 old-policy 似然意味着显存和计算开销直接下降，全 on-policy 场景也因此可行。相比 GRPO 及其各种裁剪变体，LGPO 在不同训练模式下表现更稳健。
Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?
Reinforcement learning (RL) has become central to post-training large language models (LLMs). However, popular RL methods like GRPO incur non-negligible overhead by computing both old-policy and current-policy likelihoods to form importance sampling ratios. In this work, we propose Likelihood-Gated Policy Optimization (LGPO), which enforces a soft trust region constraint via likelihood-based gating, eliminating the need to compute old-policy likelihoods. Empirically, we show that removing the importance sampling correction term does not harm training stability, whereas removing the trust region mechanism can lead to collapse. Moreover, ratio-based clipping can fail in fully on-policy training: the importance ratio stays at 1, so the ratio-based trust region constraint never activates. Under standard training settings where GRPO is stable, LGPO achieves comparable training stability and peak performance while reducing training time by ~18% on average relative to a standard implementation that recomputes old-policy likelihoods. In fully on-policy training, where GRPO fails, LGPO remains stable, enabling more efficient and robust LLM RL post-training across training regimes.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LGPO, GRPO, 强化学习, LLM后训练.
Source attribution: curated from arXiv cs.AI (@Shuibai Zhang, Junhyuck Kim, Gyeongman Kim, Jaewoong Cho).