arXiv 论文提出知识蒸馏数学模型：收益来自优化加速而非更好泛化
Two mathematical models of knowledge distillation
用数学证明蒸馏为什么有效：好处在优化更快，不在泛化更好。训练广义线性模型和做矩阵近似时能直接套用这些结论，做模型压缩的工程师和理论方向研究者都值得看。
arXiv 论文 2611.17230v1 提出并分析了知识蒸馏的数学模型，结论是蒸馏的收益来自更容易优化的损失地形，而非得到更强的模型。
作者在广义线性模型上用随机梯度下降证明：蒸馏训练的学生模型在渐近意义上比非蒸馏模型更快拟合。在秩-1 矩阵近似任务中，论文刻画了目标矩阵满足什么条件时，蒸馏下的梯度下降严格快于直接优化监督目标。
真实数据集上的实验与理论预测一致。这项工作把蒸馏的好处明确划定为优化速度，而非泛化能力，对理解为何蒸馏有效提供了理论基础。
Two mathematical models of knowledge distillation
Many hypotheses compete to explain the successes of knowledge distillation. To help address this, we propose and analyze a mathematical model of distillation, which suggests that distillation's performance comes not from obtaining better models but from easier to optimize landscapes. For generalized linear models trained with stochastic gradient descent, we prove that distillation fits performant student models asymptotically more quickly than non-distilled models. In rank-1 matrix approximation, we characterize conditions on the target matrix under which gradient descent with distillation converges strictly faster than training on the supervised objective. The theory helps delineate the ways distillation provides benefits (i.e., in optimization speed, not in generalization), and experiments on real datasets corroborate the theoretical predictions.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include 知识蒸馏, 模型压缩, 优化理论, arXiv.
Source attribution: curated from arXiv cs.LG (@Audrey Xie, Ludwig Schmidt, John Duchi).
