仅需0.1%-1%token：IER改进稀疏在线蒸馏的梯度估计
1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation
蒸馏最烧token，这篇把教师监督压到0.1%-1%还不掉点，reverse-KL目标也保留，做训练的可以直接抄代码。
arXiv 论文研究稀疏在线蒸馏（OPD）中的梯度估计问题：教师监督只分配给学生生成轨迹中一小部分token，但用采样的下一个token估计梯度会产生噪声。作者在信息几何框架下分析固定前缀处的估计误差，提出基于信噪分解的信息效率比（IER），刻画最优标量基线下的相对梯度估计误差。借助候选集近似，IER 可与现有有用性分数组合进行token选择，同时保留采样的 reverse-KL 训练目标。在数学与医疗推理任务上，加入 IER 后现有选择器在多种设置下均有提升，0.1%-1% 的稀疏配置即可达到或超过无token选择的全量 OPD，代码已在 GitHub 开源。
1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation
Sparse on-policy distillation (OPD) allocates teacher supervision to a small subset of tokens in student-generated trajectories. However, useful teacher guidance can yield a noisy update when its gradient is estimated from a sampled next token. We study this estimation problem at a fixed prefix in information geometry and propose an information-efficiency ratio (IER) based on a signal-to-noise decomposition. IER characterizes relative gradient estimation error under an optimal scalar baseline. A candidate-set approximation enables token selection based on IER and its combination with existing usefulness scores, while retaining the sampled reverse-KL training objective. On mathematical and medical reasoning tasks, adding IER improves existing selectors in multiple settings, with sparse configurations matching or exceeding full OPD without token selection at small token budgets of 0.1\%--1\%. These results support accounting for both usefulness and gradient-estimation reliability when allocating sparse supervision. Our code is available at https://github.com/BruceSheng1202/IER-OPD.
