这篇论文揭示了OPD更新的稀疏性和几何特性，对做模型蒸馏和微调的团队有直接参考价值——你可以只训练关键子网络来节省计算，同时理解为什么AdamW比SGD更优。建议点开看看实验细节。
该论文分析了在线策略蒸馏（OPD）在语言和多模态模型中的参数更新特性，发现OPD更新具有稀疏性，主要集中在FFN层，且仅训练这些子网络即可接近完整OPD的性能。更新在几何上是满秩但谱集中的，主要落在源权重接近零的坐标上。研究还发现，密集教师监督下SGD优化器不如AdamW，因为AdamW的自适应缩放对保持异构梯度尺度更有效。这些结果表明，OPD并非简单的密集参数重写，而是保留了策略后训练的几何特征。
Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation
On-policy distillation (\textsc{OPD}) has recently become a prominent post-training recipe as it combines two desirable ingredients: on-policy student trajectories and dense teacher supervision, yet how this hybrid changes a model's parameters remains unclear. Across several language and vision-language model pairs and use cases, our analysis yields two main findings. On sparsity, \textsc{OPD}-style updates are small and coordinate-sparse. They are distributed across layers and are usually FFN-heavy. This sparse structure is operationally useful: training only the discovered subnetwork recovers nearly the same performance as full \textsc{OPD}. However, the sparsity-inducing SGD optimizer underperforms AdamW in our optimizer ablation, likely because dense teacher supervision preserves heterogeneous coordinate-wise gradient scales where AdamW's adaptive scaling remains useful. On geometry, the updates are numerically full-rank but spectrally concentrated; they lie mostly away from the principal singular subspaces of the source weights and fall disproportionately on coordinates where the source weights are close to zero. These findings suggest that dense teacher supervision does not turn \textsc{OPD} into ordinary dense parameter rewriting; instead, \textsc{OPD} retains important geometric signatures of on-policy post-training.