This research reveals that on-policy distillation's gains come primarily from algorithm improvements rather than data volume, allowing dramatic reduction in training requirements. Developers can implement One-Shot OPD to train models more efficiently, with potential cost savings and faster iteration cycles for AI companies in both China and the US.
CompassOPD enables better knowledge transfer between different AI model families, potentially improving reasoning capabilities in existing models without full retraining.
The findings suggest OPD can achieve near-full performance with minimal data, potentially reducing training costs for AI developers in ASEAN and beyond.
This research offers a practical training recipe that improves LLM reasoning performance, with clear switching signals between stages applicable to developers working on reasoning models.
OPSA improves Avg@32 by 35.41 points on AIME24 over Qwen3-1.7B, offering a supervision-free alternative to OPD that outperforms the original method.
发了个叫Adaptive FastOPD的蒸馏加速方法，比OPD 15K快一半多，性能还更高。做训练加速的可以看看。
这篇用一个可控环境拆解了AI长程规划的训练机制，发现次优数据会拖累性能，多教师蒸馏能整合不同能力，值得做训练方法的人细读。
Miles 发布了 OPD 方法，让 Qwen3.5 模型推理更快更准，无需任务奖励，比传统蒸馏更高效，适合模型压缩场景。
这篇论文揭示了OPD中后面token质量差的问题，提出IW-OPD加权方案，在AIME-2025上提分6.9，做蒸馏研究的值得一看。
这篇论文揭示了OPD更新的稀疏性和几何特性，对做模型蒸馏和微调的团队有直接参考价值——你可以只训练关键子网络来节省计算，同时理解为什么AdamW比SGD更优。建议点开看看实验细节。
