ST-OPSD：用激活转向修复思维模型自蒸馏中的教师缺陷
Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models
自蒸馏训练推理模型会让学生学会瞎编参考、不敢回头检查，这篇用激活转向治好了这两个毛病，AIME、HMMT 上准确率也涨了，做训练的可以看看。
论文指出 on-policy self-distillation（OPSD）在训练思维模型时存在两类失败模式：reference hallucinations（学生引用不存在的参考来为推理步骤辩护）和 reconsideration suppression（教师抑制学生在不确定步骤的验证与修正）。作者提出 Steered-Teacher OPSD（ST-OPSD），从模型生成的推理轨迹中提取两个方向，在计算蒸馏目标时施加到教师隐藏状态上，抑制引用参考行为并鼓励重新审视，同时用 marker-family calibration。学生在未被转向的情况下从教师分布中学习。在 AIME 和 HMMT 上，ST-OPSD 的准确率超过原始 OPSD 和其他干预方法，并将重新审视率与参考幻觉率恢复到基础模型水平。
Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models
On-policy self-distillation (OPSD) trains a student on its own reasoning trajectories using token-level supervision from a reference-conditioned teacher initialized from the same model. However, access to the reference can induce teacher behaviors that undermine independent reasoning in thinking models, leading to reference hallucinations, where the student cites an unavailable reference to justify reasoning steps, and reconsideration suppression, where the teacher discourages verification and revision at uncertain steps. We introduce Steered-Teacher OPSD (ST-OPSD) to jointly address these failure modes through activation steering. ST-OPSD extracts two directions from model-generated reasoning trajectories and applies them to the teacher's hidden states during distillation target computation, suppressing reference-invoking behavior and encouraging reconsideration with marker-family calibration. The student learns from the resulting teacher distributions without being steered. Experiments on AIME and HMMT show that ST-OPSD improves accuracy over vanilla OPSD and other interventions while restoring reconsideration and reference-hallucination rates to base-model levels.
