做 LLM 推理优化的研究者可以关注——SGSD 用技能库替代参考答案作为先验，降低了蒸馏对标注数据的依赖，数学推理场景效果显著，值得在自蒸馏框架中尝试。
本文提出 Skill-Conditioned Gated Self-Distillation (SGSD) 方法，用于改进大语言模型的推理能力。传统自蒸馏方法依赖可信的先验信息（如参考答案），而 SGSD 从经验技能库中检索技能-错误对，构建多教师池，通过验证器判断教师极性，并设计门控目标函数来蒸馏有效信息。在多个数学推理基准上，SGSD 在 Qwen3-1.7B 上平均比 GRPO 提升 6.2%，比 OPSD 提升 1.7%，且对先验信息的假设更弱。代码已开源。
Skill-Conditioned Gated Self-Distillation for LLM Reasoning
On-policy self-distillation (SD) improves LLM reasoning by using teacher-side privileged information (PI) to turn sparse verifier outcomes into dense token-level supervision. Existing methods usually assume trusted PI, such as reference answers or successful traces. We ask whether PI can instead come from an experience-derived skill bank, where retrieved skills are compact and reusable but may also be irrelevant or misleading. We propose Skill-Conditioned Gated Self-Distillation (SGSD), which formulates skill-based SD as teacher hypothesis validation rather than unconditional imitation. SGSD retrieves skill-mistake pairs, constructs a multi-teacher pool, and lets all skill-conditioned teachers score the same plain-prompt student rollout. The verifier validates each teacher's polarity: supporting a success or suppressing a failure gives positive supervision, while the opposite stance is reversed. A robust gated objective then distills informative teacher-student disagreements while suppressing uncertain or extreme signals. Experiments on multiple mathematical reasoning benchmarks show that SGSD consistently improves over GRPO and remains competitive with answer-conditioned OPSD under a weaker PI assumption. For example, on Qwen3-1.7B, SGSD outperforms GRPO by 6.2% and OPSD by 1.7% on average on AIME24, AIME25, and HMMT25. Our code is available at https://github.com/walawalagoose/SGSD.