组合优化任务通常依赖大型模型，但部署成本高。本文给出了理论保证，让做图神经网络和算法对齐的开发者知道何时可以安全地蒸馏到小模型，值得关注。
本文研究了在组合优化任务中，如何将大型模型的知识蒸馏到更小、更高效的模型。作者假设目标模型是图神经网络，其架构与任务的动态规划算法对齐。基于决策树蒸馏的最新理论分析，论文证明了当源模型足够丰富（通过线性表示假设形式化）时，蒸馏问题可以在动态规划转移函数的复杂度参数内高效解决。该工作为算法对齐框架下的成功蒸馏提供了严格充分条件。
Towards Distillation Guarantees under Algorithmic Alignment for Combinatorial Optimization
Distillation transfers knowledge from a large model trained on broad data to a smaller, more efficient model suitable for deployment. In structured prediction settings, prior knowledge about the task can guide the choice of a target architecture that is algorithmically aligned with the underlying problem. Building on recent learning-theoretic analyses of decision-tree (DT) distillation (Boix-Adsera, 2024), we study when distillation succeeds for combinatorial optimization tasks. We focus on the case where the target model is a graph neural network whose architecture is aligned with a dynamic programming (DP) algorithm for the task. Assuming that the source model is sufficiently rich, formalized through the linear representation hypothesis (LRH) (Elhage et al., 2022; Park et al., 2024), we show that the distillation problem can be solved efficiently in the complexity parameters of the DP transition function, represented as a DT. Our results provide a rigorous sufficient condition for successful distillation in the flavour of algorithmic alignment.