这篇论文用几何视角解释了神经网络为什么先死记硬背后突然泛化，还找到了一个简单技巧（约束隐藏层半径）让 grokking 快 6 倍，值得搞训练的人看看。
该论文对神经网络在算法任务上的延迟泛化（grokking）现象进行几何分析，发现交叉熵优化下隐藏表示的径向膨胀是延迟的主因。作者提出径向-角分解并推导三个可检验命题，引入单一超参数范数惩罚将激活约束在 sqrt(d) 半径超球面上。在模算术任务上，该惩罚使 MLPs 和 Transformers 的 grokking 加速高达 6 倍；对 10M 参数的 nanoGPT 在 3 位数加法上训练步数减半。
Radial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed Generalization
Why do neural networks memorize algorithmic training data long before they generalize? We present a geometric case study demonstrating that, on tasks where generalization requires discovering structured low-dimensional circuits, the memorization-generalization delay is driven by radial inflation of hidden representations under cross-entropy optimization. We formalize a radial-angular decomposition of activation-space dynamics and derive three testable propositions: (i) that penalizing radial inflation induces anisotropic, data-dependent weight regularization; (ii) that it suppresses radial gradient energy below the isotropic random baseline, forcing predominantly angular updates; and (iii) that it biases convergence toward flatter minima. To empirically validate these propositions, we study a single-hyperparameter norm penalty that softly constrains activations to a sqrt(d)-radius hypersphere. On modular arithmetic, this penalty accelerates grokking up to 6x across MLPs and Transformers, and halves training steps for a 10M-parameter nanoGPT on 3-digit addition.