这篇论文揭示了自蒸馏方法的一个隐藏缺陷：虽然准确率不错，但多样性会变差，导致复杂推理场景下失效。做RL或推理模型的人值得看看。
一篇论文研究了基于采样示范的在线自蒸馏方法对输出多样性的影响。该方法通过单一模型同时作为教师和学生，教师以正确示范为条件提供密集的token级反馈，在pass@1准确率上表现优异。但论文发现，这会导致推演多样性降低，pass@k曲线变平，即增加推演次数无法提升准确率。作者将原因追溯到自蒸馏设计中的复合偏差：教师在对学生推演评分时以采样到的正确推演为条件，通过模型自身偏见传导反馈。在可控的图路径发现任务和科学问答基准上，自蒸馏模型在平均性能上与强化学习相当或更优，但功能和语义多样性显著下降，在需要多样化策略的分布外场景中失败。
On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student, with the teacher conditioned on a correct demonstration to provide dense token-level feedback. We show that this could come at a hidden cost: rollout diversity decreases and pass@k curves flatten (i.e., generating more rollouts fails to improve accuracy). We trace this to compounding biases in the design of self-distillation with sampled demonstrations. The teacher scores each student rollout while conditioned on a sampled correct rollout, channeling its feedback through the model's own biases. We theoretically analyze the optimal self-distillation policy and show that it tilts the base distribution by a pointwise conditional mutual information score between the student's rollout and the correct rollout used as context. Unlike the ideal optimal on-policy reinforcement learning (RL), which preserves probability ratios among equally correct rollouts, self-distillation can amplify existing probability gaps, concentrating mass on already-dominant modes. On a controlled graph path-finding task and science question-answering benchmarks, self-distilled models match or exceed RL on average performance but exhibit substantially lower functional and semantic diversity, failing on out-of-distribution settings that require diverse strategies.