论文精选

学习动力学揭示权重诱导的层级Gram度量层次结构

Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics

精选理由

该研究为深度网络学习动力学提供了新的理论框架,做深度学习理论或理解网络内部机制的读者可以直接参考其层级Gram度量方法。

AI 摘要

该研究探讨了具有固定读出层和二次损失的前馈ReLU网络,旨在将梯度下降重写为训练集空间上定义的场的集体动力学,而非权重空间的动力学。对于单隐层网络,可以从激活动力学中消除权重变量,得到残差的封闭方程,该方程由输入几何矩阵和动态共激活矩阵分解的集体核控制。对于更深网络,残差动力学保留了清晰的层级核结构,但从三层深度开始,封闭需要一组权重诱导的Gram算子层次结构来跨层传输信息。这项工作为理解深度网络的学习动态提供了新的理论视角。

原文 · arXiv cs.LG

Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics

We study feed-forward ReLU networks with fixed readout and quadratic loss. The aim is to rewrite gradient descent not primarily as a dynamics in weight space, but as a collective dynamics closed in terms of fields defined on the training-set space. For a single hidden layer, the weight variables can be eliminated from the activation dynamics, yielding a closed equation for the residuals governed by a collective kernel that factorizes into an input-geometric matrix and a dynamical co-activation matrix. For deeper networks, the residual dynamics retains a clean layer-wise kernel structure. However, from depth three onward, closure requires a hierarchy of weight-induced Gram operators that mediate information transport across layers.

学习动力学揭示权重诱导的层级Gram度量层次结构 · AI 热点