想知道GELU、多头注意力这些架构背后为什么有隐式偏好？这篇论文给出了严格的理论解释。
该论文提出了一个统一框架，用于刻画现代神经网络架构（如使用GELU、SiLU、SwiGLU激活的前馈网络、具备正弦和旋转位置编码的多头注意力、以及多种门控设计的混合专家模型）中的梯度下降守恒定律。此前守恒定律仅在线性和ReLU网络中被理解。实验验证了所预测的不变量。
Conservation Laws for Modern Neural Architectures
Understanding gradient descent dynamics is key to explaining the success of over-parameterized models, where implicit bias manifests through conservation laws in gradient flow. While such laws are well understood for linear and ReLU networks, they remain largely unexplored for modern architectures. This work develops a unified framework to characterize conservation laws for contemporary models, including feedforward networks with GELU, SiLU, and SwiGLU activations, multihead attention with sinusoidal and rotary positional encodings, and Mixture-of-Experts architectures under diverse gating designs. Our theoretical findings are supported by experiments that validate the predicted invariants.