做LLM RL后训练的团队终于有了更优雅的信任区域控制方案——DRPO用连续正则化替代硬裁剪，解决了长尾词汇下梯度丢弃问题，训练更稳且收敛更快，做RLHF或GRPO优化的建议直接读论文。
该论文指出，在LLM后训练中常用的PPO和GRPO方法依赖重要性比率裁剪来约束策略更新，但在长尾词汇分布下，比率无法准确反映分布偏移。DPPO虽改用散度掩码，但硬掩码会直接丢弃越界token的梯度，导致信息损失。作者提出DRPO，用平滑的优势加权二次正则项替代硬掩码，在保持相同信任区域几何的同时，提供连续梯度权重，既能抑制发散更新，又能在边界外提供修正信号。实验表明，DRPO在不同模型规模、架构和精度设置下均提升了训练稳定性和效率。
Rethinking the Divergence Regularization in LLM RL
Reinforcement learning (RL) has become a key component of post-training large language models (LLMs). In practice, LLM RL is often off-policy because of training-inference mismatch and policy staleness, making trust-region control essential for stable optimization. Mainstream methods such as PPO and GRPO approximate this control with a ratio-clipping mechanism, but the importance ratio can be a poor proxy for distributional shift in long-tailed vocabularies. Recent work such as DPPO addresses this mismatch by replacing ratio-based clipping with a divergence-based mask, yielding a trust region defined by the sampled token's absolute probability shift. However, DPPO still relies on a hard mask: once a token crosses the trust-region boundary in a harmful direction, its gradient is discarded rather than corrected. To address this, we propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift. DRPO preserves the same trust-region geometry as DPPO while inducing bounded, continuous gradient weights that attenuate diverging updates and provide corrective signals beyond the boundary. Experiments across model scales, architectures, and precision settings show that DRPO improves the stability and efficiency of LLM RL training.