SimOPD提出通过相似度分布做在线策略蒸馏
SimOPD: Relational Knowledge Transfer for On-Policy Distillation
一种新的蒸馏方法，不看 token 概率也不硬对齐隐藏状态，靠响应相似度传递教师知识，跨架构蒸馏也能涨点，做模型压缩的可以看看。
SimOPD 针对 on-policy distillation 中监督信号不足的问题，借鉴对比学习思路，用学生与教师各自的记忆库计算响应相似度分布来传递教师内部知识。学生在自己的表示空间内匹配这些分布，避免了隐藏状态直接匹配所需的跨模型投影。实验显示，在同架构和跨架构设置下，SimOPD 比最强基线分别提升 2.6 和 3.1 个百分点。
SimOPD: Relational Knowledge Transfer for On-Policy Distillation
A key challenge in on-policy distillation (OPD) is extracting effective supervision from teacher knowledge. Supervision from token probabilities may underutilize internal knowledge, while direct hidden-state matching forces the student to shift to the teacher's feature space and complicates cross-architecture transfer. Inspired by contrastive learning's use of relationships between instances, we propose \textbf{SimOPD} to transfer teacher internal knowledge through response similarities. Specifically, SimOPD uses paired memory banks of student-generated responses to extract supervision from the teacher's internal representations as similarity distributions. The student then matches these distributions within its own representation space, retaining representational freedom and removing the need for cross-model projection. Extensive experiments show that SimOPD improves the overall average over the strongest baselines by up to 2.6 and 3.1 percentage points in same- and cross-architecture settings.
