这篇论文揭示了OPD中后面token质量差的问题，提出IW-OPD加权方案，在AIME-2025上提分6.9，做蒸馏研究的值得一看。
标准在策略蒸馏（OPD）对所有token均匀加权，但研究发现学生rollout越长，后续token与教师分布偏差越大。仅使用前30%的token即可达到全token性能，而仅用后30%几乎学不到东西。基于约束优化视角，提出重要性加权在策略蒸馏（IW-OPD），根据累积偏差动态调整token权重。IW-OPD收敛更快，在AIME-2025基准上性能提升多达6.9分。
On the Position Bias of On-Policy Distillation
On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers. In the standard KL objective of OPD, token-level losses are uniformly averaged, implying equal weights for all tokens. However, we discover that not all tokens are created equal: as student rollouts grow longer, they deviate further from the teacher's distribution, leading to degraded supervision quality at later positions. As a result, OPD using only the first 30% of tokens can perform comparably to using all tokens, whereas OPD using only the last 30% of tokens barely learns anything. In this work, we provide a principled understanding of this issue through the lens of constrained optimization. Based on these insights, we derive Importance-Weighted On-Policy Distillation (IW-OPD), in which the weight assigned to each token depends on the accumulated discrepancy between the student's and teacher's distributions, naturally upweighting earlier tokens and downweighting later ones with larger deviations. We show that IW-OPD converges significantly faster than OPD, with better learning efficiency, and achieves better final performance than standard OPD in both same-size and cross-scale settings, improving performance up to 6.9 points on AIME-2025.