不需要标注和外部反馈，U-OPSD靠投票让Qwen3数学成绩再涨一截，追平甚至超过OPSD和GRPO。
U-OPSD 提出只用模型自身生成做 on-policy 自蒸馏，不依赖真实标签、环境反馈或更大模型。它通过多次采样后多数投票构造伪答案，并把教师分布建立在最短伪答案上，蒸馏到模型最长错误补全的前缀。在 AIME24、AIME25、HMMT25、MATH500、AMC23 上，U-OPSD 让 Qwen3 4B/8B 非思考模式相对基座提升 8.5%/10.7%，平均超过 OPSD 3.2%/2.3%。思考模式下，U-OPSD 在 4B 超 OPSD 0.9%，8B 持平，并分别超 GRPO 0.7%/1.1%。
On-Policy Self-Distillation without Any Supervision
On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose Unsupervised On-Policy Self-Distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo-solution by majority vote under a self-consistency threshold. It then conditions a teacher distribution on the shortest pseudo-solution and distills it into prefixes of the model's longest incorrect completion, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT), such as OPSD and GRPO. On AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at the 4B and 8B scales, respectively, and outperforms OPSD by an average of 3.2% and 2.3%. In thinking mode, U-OPSD remains on par with OPSD, outperforming it by 0.9% at 4B and matching it at 8B, while surpassing GRPO by 0.7% and 1.1%, respectively.