#online distillation

This paper presents a new parameterization for per-token gating in on-policy knowledge distillation, which is a key technique for training smaller models to mimic larger ones. It provides a framework for comparing different gating designs.

这篇论文讲了个反直觉的事：蒸馏时师生token一致反而可能有害。TIDE方法在数学推理上把准确率从6.9%拉到20.3%，还让回答短了3.6倍，值得做LLM后训练的人看看。