#在线蒸馏

This paper presents a new parameterization for per-token gating in on-policy knowledge distillation, which is a key technique for training smaller models to mimic larger ones. It provides a framework for comparing different gating designs.

#FKL/RKL #distillation #Qwen3-32B #Qwen3-4B

arXiv cs.AI@Suwan Wu et al. (4)Source

Aug 11

PapersSelected11:39

Chinese contentTIDE：解决LLM在线蒸馏中师生不匹配问题

这篇论文讲了个反直觉的事：蒸馏时师生token一致反而可能有害。TIDE方法在数学推理上把准确率从6.9%拉到20.3%，还让回答短了3.6倍，值得做LLM后训练的人看看。

#TIDE #Qwen3 #math reasoning

arXiv cs.AI@Zichao Yu et al. (7)Source

Aug 3

Papers09:28

Chinese contentAdaptive FastOPD：进度感知的滚动视野扩展实现高效在线蒸馏

发了个叫Adaptive FastOPD的蒸馏加速方法，比OPD 15K快一半多，性能还更高。做训练加速的可以看看。

#Adaptive FastOPD #OPD #training acceleration

arXiv cs.LG@Qian Tan et al. (5)Source