这项研究为深度学习从业者提供了一种无需数据即可监控过拟合的新工具，尤其适合训练长期 grokking 模型的团队。建议关注模型泛化能力的开发者点开，了解如何用随机矩阵理论提前发现模型退化信号。
研究者提出一种基于随机矩阵理论的新方法，无需访问训练或测试数据即可检测深度学习模型的过拟合。该方法通过随机化每层权重矩阵，拟合 Marchenko-Pastur 分布，识别出违反自平均性的异常值（称为 Correlation Traps）。在长期 grokking 过程中，过拟合的“anti-grokking”阶段会出现这些陷阱，其数量和规模随测试准确率下降而增长。研究还提供了区分良性陷阱与有害陷阱的实证方法，并发现部分基础大语言模型也存在类似陷阱，暗示潜在的有害过拟合。
Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory
Training Neural Networks (NNs) without overfitting is difficult; detecting that overfitting is difficult as well. We present a novel Random Matrix Theory method that detects the onset of overfitting in deep learning models without access to train or test data. For each model layer, we randomize each weight matrix element-wise, $\mathbf{W} \to \mathbf{W}_{\mathrm{rand}}$, fit the randomized empirical spectral distribution with a Marchenko-Pastur distribution, and identify large outliers that violate self-averaging. We call these outliers Correlation Traps. During the onset of overfitting, which we call the "anti-grokking" phase in long-horizon grokking, Correlation Traps form and grow in number and scale as test accuracy decreases while train accuracy remains high. Traps may be benign or may harm generalization; we provide an empirical approach to distinguish between them by passing random data through the trained model and evaluating the JS divergence of output logits. Our findings show that anti-grokking is an additional grokking phase with high train accuracy and decreasing test accuracy, structurally distinct from pre-grokking through its Correlation Traps. More broadly, we find that some foundation-scale LLMs exhibit the same Correlation Traps, indicating potentially harmful overfitting.