学习动力学揭示权重诱导的层级Gram度量层次结构
Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics
该研究为深度网络学习动力学提供了新的理论框架，做深度学习理论或理解网络内部机制的读者可以直接参考其层级Gram度量方法。
该研究探讨了具有固定读出层和二次损失的前馈ReLU网络，旨在将梯度下降重写为训练集空间上定义的场的集体动力学，而非权重空间的动力学。
对于单隐层网络，可以从激活动力学中消除权重变量，得到残差的封闭方程，该方程由输入几何矩阵和动态共激活矩阵分解的集体核控制。对于更深网络，残差动力学保留了清晰的层级核结构，但从三层深度开始，封闭需要一组权重诱导的Gram算子层次结构来跨层传输信息。
这项工作为理解深度网络的学习动态提供了新的理论视角。
Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics
We study feed-forward ReLU networks with fixed readout and quadratic loss. The aim is to rewrite gradient descent not primarily as a dynamics in weight space, but as a collective dynamics closed in terms of fields defined on the training-set space. For a single hidden layer, the weight variables can be eliminated from the activation dynamics, yielding a closed equation for the residuals governed by a collective kernel that factorizes into an input-geometric matrix and a dynamical co-activation matrix. For deeper networks, the residual dynamics retains a clean layer-wise kernel structure. However, from depth three onward, closure requires a hierarchy of weight-induced Gram operators that mediate information transport across layers.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include 深度学习理论, 学习动力学, Gram度量, ReLU网络.
Source attribution: curated from arXiv cs.LG (@Claudio Nordio).