OPSA方法比OPD提升16.77分，在AIME24上相对增益263%，无需教师监督。
研究人员分析了在线策略蒸馏(OPD)中的教师监督，发现存在大量噪声，且噪声随教师规模增加而增加。学生策略对这种噪声不敏感，无论保留还是移除噪声监督，性能相当。研究发现OPD主要通过抑制低概率token来工作，这不需要教师监督。基于此，研究人员提出了无监督的在线策略自适应(OPSA)方法，在AIME24基准上将Avg@32提高了35.41点。
Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base \texttt{Qwen3-1.7B}, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.