OPO：按样本自适应裁剪比率，改进 PPO 的固定裁剪策略
Ordered Policy Optimization
一篇 RL 论文，作者给 PPO 的固定裁剪换了玩法：每个 minibatch 里排序挑出极端比率，单独给自适应半径，MuJoCo 和 Atari 上跑赢了基线。
PPO 用固定裁剪范围处理新旧策略之间的重要性比率，无法区分有害的极端比率和可容忍的大幅变化。论文提出 Ordered Policy Optimization（OPO），在每个 minibatch 内对重要性比率排序，选出影响最大的极端比率，并根据预设的尾部轮廓为其分配自适应的逐样本裁剪半径。改进后的目标函数仍保持一阶形式，只需 minibatch 排序和修改策略损失即可实现。在 MuJoCo 和 Atari 基准上，OPO 相对既有基线取得了更强的性能。
Ordered Policy Optimization
Proximal Policy Optimization (PPO) stabilizes policy updates by clipping the importance ratio between the new policy and the data-collecting policy. The standard rule uses one fixed clipping range for all samples. While this choice offers simplicity and effectiveness, it fails to distinguish statistically harmful extreme ratio values from large, yet tolerable changes. We study how to adapt the clipping radius to the reliability of the induced importance ratio distribution. Our case study demonstrates that reliable updates require controlling these extreme ratios, rather than merely verifying that individual ratios fall within a fixed interval. Motivated by this observation, we propose Ordered Policy Optimization (\textsc{OPO}). In each minibatch, \textsc{OPO} sorts the importance ratios, selects the extreme ratios that potentially have the largest effect on the update, and assigns them adaptive sample-wise radii instead of using the fixed clipping radius. These radii are derived from a prescribed tail profile, which specifies desired target values for the selected extreme ratios. The resulting objective remains first-order and requires only minibatch sorting and a modified policy loss. Experiments conducted on MuJoCo and Atari benchmarks demonstrate that this simple modification achieves strong performance relative to established baselines.
