二次神经网络中宽度与数据如何塑造泛化缩放定律
How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
这篇论文给出了二次网络里参数和样本数怎么影响泛化误差的数学公式，比单纯看数据或算力更深入。
该论文在二次两层神经网络中，通过ℓ2正则化经验风险最小化，分析了泛化误差随参数数量、样本量和宽度变化的精确表达式。
研究发现泛化误差遵循依赖于目标谱结构的数据驱动幂律，并揭示了不同缩放区域间的相图转变，包括插值起始点的特征。实验基于有限样本和结构化数据，为理解特征学习模型的缩放行为提供了理论框架。
How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
Understanding how performance scales jointly with model size and data is a central problem in modern machine learning. Existing theoretical works on scaling laws typically describe generalization as a function of data or compute, often in fixed-feature or infinite-width regimes and for online SGD. Here, we instead study how generalization scales with the number of trainable parameters and the number of samples in a feature-learning model. We analyze $\ell_2$-regularized empirical test error minimization in a quadratic two-layer network in a finite-sample setting with structured data. This setting allows for an explicit characterization of the generalization error as a function of the number of samples, model width, and regularization. Our results reveal a phase diagram with distinct scaling regimes as the number of parameters varies. In particular, the generalization error follows data-dependent power laws controlled by the spectral structure of the target. We further characterize the transitions between regimes, including the onset of interpolation, and their impact on generalization.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include Quadratic Neural Networks, 泛化缩放定律, 相图, 正则化.
Source attribution: curated from arXiv cs.AI (@Julius Girardin, Emanuele Troiani, Yizhou Xu, Vittorio Erba, Florent Krzakala, Lenka Zdeborová).
