This review delves into the challenges of On-Policy Self-Distillation, highlighting the 'collapse' issue and offering insights into its causes. It's a must-read for those interested in understanding the nuances of model training with privileged information.
On-Policy Self-Distillation (OPSD) removes the need for a second model as teacher, using the model itself with privileged information. Early results show accuracy comparable to reinforcement learning. However, the review identifies 'collapse' as a dominant failure mode, which is exacerbated by privileged information. The review analyzes collapse through three factors: signal application, teacher's privileged information, and signal change dynamics. The study focuses on mathematical reasoning, with no new experiments reported. The contribution is a shared vocabulary for discussing phenomena and clarifying settled and disputed points.
On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.