Miles 发布了 OPD 方法，让 Qwen3.5 模型推理更快更准，无需任务奖励，比传统蒸馏更高效，适合模型压缩场景。
Miles 框架新增 On-Policy Distillation（OPD）作为一等训练原语。在 Qwen3.5-35B-A3B 上，纯 OPD 使推理 token 数从 18.6k 降至 5.5k–6.7k，缩短约 3 倍，同时 held-out DAPO 准确率从 0.8457 升至 0.8945。反向 KL 散度从 0.045 降至 0.010，学生模型收敛至教师。采用稀疏逐位置评分代替密集 O(R²K) 评分，提升效率。支持纯蒸馏或与 GRPO/PPO 奖励结合，计划扩展多教师和 RL 场景。
🚀 New Miles blog: OPD Support in Miles On-Polic…
🚀 New Miles blog: OPD Support in Miles
On-Policy Distillation (OPD) is now a first-class training primitive in Miles. In our first experiment, pure OPD made Qwen3.5-35B-A3B reason ~3X shorter while slightly improving accuracy using no task reward at all, only the teacher's guidance.
Highlights from a controlled self-distillation run: 1️⃣ Accuracy improved: held-out DAPO rose from 0.8457 to 0.8945 2️⃣ ~3X shorter responses: rollout length dropped from ~18.6k tokens to 5.5k–6.7k after one update 3️⃣ Student converged to teacher: reverse-KL fell from 0.045 to 0.010 4️⃣ Efficient scoring: sparse per-position teacher scoring instead of dense O(R²K) payload 5️⃣ Flexible to use: can run pure distillation, or OPD + GRPO/PPO rewards
What's next: OPD-augmented RL, scaling studies, multi-teacher OPD, and more training recipes. Special thanks to @nvidia for the collaboration; everything ran colocated on a single 8×B200 node.
- arXiv: DeepSeek07-22 12:45原文