这篇论文把OPSD训练时模型思维崩溃的原因和解决方法都讲清楚了，AD-OPSD在数学题上能提4个点，做推理优化的值得看看。
本文发现On-Policy Self-Distillation (OPSD)在复杂推理任务中会导致“Thinking Collapse”，即模型中间推理行为（以epistemic-token密度衡量）显著下降。通过熵梯度分析和token级目标分解，确定该崩溃源于高熵决策点的激进教师梯度。提出自适应双视角OPSD (AD-OPSD)，通过冻结基础模型参考先验锚定高风险token。在多个数学基准（如GSM8K、MATH）上，AD-OPSD相比标准OPSD平均准确率提升最高达+4.1%。
Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation
On-Policy Self-Distillation (OPSD) has emerged as a crucial paradigm for enhancing and aligning Large Language Models (LLMs). However, in complex reasoning tasks, OPSD paradoxically degrades downstream performance. In this paper, we systematically investigate this pathology and identify a severe optimization trap we define as \textbf{Thinking Collapse} -- a sharp decline in the model's native intermediate reasoning behavior, measured by epistemic-token density (ET per 1k). Through entropy-based gradient masking and token-level target analysis, we show that this collapse is triggered by aggressive teacher gradients at high-student-entropy decision forks, where student epistemic tokens are frequently suppressed into teacher non-epistemic targets and are highly concentrated in high pointwise student-teacher divergence regions. To resolve this optimization pathology, we propose \textbf{Adaptive Dual-Perspective OPSD (AD-OPSD)}, a robust control framework that dynamically moderates the self-distillation objective. AD-OPSD selectively anchors high-suppression-risk sandboxed tokens to a reference prior derived from the frozen base model via an asymmetrical pointwise divergence gate, preserving native thinking capacity while retaining OPSD's error-correcting power. Extensive experiments across competitive mathematical benchmarks show that AD-OPSD improves over standard OPSD by up to \textbf{+4.1\%} absolute average accuracy across diverse model scales and datasets. Further analysis demonstrates that AD-OPSD mitigates thinking collapse and generalizes robustly to different post-training paradigms.