这篇论文提出了一种更聪明的掩码方法，解决了PPO和DPPO在方向判断上的不一致，让LLM强化学习训练更稳定高效。
PPO通常使用采样标记重要性比率来稳定LLM强化学习的离策略更新，包括邻近准则和方向准则。DPPO改进了邻近准则但方向准则仍继承自PPO，其基于重要性比率的方向可能与散度变化不一致。本文提出预测性分歧掩码，通过闭式解预测策略梯度步骤对同一散度的影响，并针对生产环境中的Top-K词汇开发了两种轻量估计器。实验表明，该方法在多个模型尺度和精度设定下提升了RL训练效果。
Predictive Divergence Masks for LLM RL
Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates. The dominant PPO-style approach uses the sampled-token importance ratio for two criteria: a proximity criterion, which asks whether the policy has moved too far from the behavior policy, and a direction criterion, which asks whether the update pushes it farther away. Recent work DPPO improves the proximity criterion by replacing PPO's ratio-based test with a probability divergence between the behavior and training policies. However, its direction criterion is still inherited from PPO. A token can be masked only when the sampled-token importance ratio moves away from one. We observe that this ratio-based direction criterion is a single-sample proxy that can disagree in sign with the change of the divergence that defines the proximity criterion. We therefore propose the predictive divergence mask, which asks whether the next policy-gradient step will increase or decrease the same divergence used by the trust region. For the discrete softmax policies used in LLM RL, we derive this prediction in closed form. Because production rollout engines expose only a truncated (top-K) view of the vocabulary, we develop two lightweight top-$K$ estimators for this prediction. Detailed analysis shows the divergence-based direction is better aligned with the realized change of the divergence than the sampled ratio, and the resulting masks improve RL training across model scales and precision settings.