这篇论文从理论层面解释了为什么过参数化模型能泛化——做高维统计学习或深度学习理论的研究者，看完会对“良性过拟合”有更精确的数学理解，值得深入阅读。
该论文研究了在高维过参数化场景下，能够完美插值训练数据的线性分类器的泛化性能分布。在两种数据生成模型（高斯混合模型和逻辑回归模型）下，作者证明了当样本数与维度之比足够小时，几乎所有插值分类器的泛化误差都集中在某个最优值附近，只有指数级小部分的分类器性能显著不同。通过数值比较，发现梯度下降和经验风险最小化等高效算法找到的插值器性能优于绝大多数随机插值器，揭示了这些算法在过参数化设置下具有非平凡的良性过拟合特性。这项工作为理解深度学习中的“良性过拟合”现象提供了理论支撑。
How abundant are good interpolators?
Let $S$ be the set of unit norm linear classifiers $θ\in \mathbb{R}^d$ which correctly classify every point of a labeled dataset $(X_i,y_i)_{i=1}^n$, $X_i \in \mathbb{R}^d$, $y_i \in \{-1,+1\}$, with a possibly negative margin $κ$ fixed in advance. Under two natural data-generating distributions of the $(X,y)$ pairs -- a Gaussian mixture model and a logistic model with Gaussian features -- and in the proportional regime $n/d \to α$ with small enough $α$, we establish a large deviation principle on the event that a point $θ$ chosen uniformly at random from $S$ achieves a given generalization error, with high probability over the choice of the data. The associated large deviation rate function is deterministic and describes the proportion, at the exponential scale in $d$, of interpolating classifiers having a given desired performance. As a consequence, we establish the following concentration phenomenon: all but an exponentially small fraction of interpolating classifiers have approximately the same generalization performance given by the unique maximizer of this rate function. We numerically compare this maximizer to the performance of empirical risk minimization by gradient descent and to the performance of a natural linear program, both finding a point in $S$, and deduce that in the overparametrized regime of small $α$, these efficient procedures outperform the vast majority of interpolators, pointing to their nontrivial benign overfitting in this setting.