理论研究者终于有了一个更精确的工具来刻画Transformer泛化——傅里叶谱视角比Rademacher复杂度更贴近实际训练行为，做深度学习理论或可解释性的同学值得细读。
该研究从布尔函数的傅里叶谱角度分析Transformer的泛化行为，提出稀疏且集中在低阶分量的频谱能构造低锐度（low-sharpness）的平坦最小值，从而获得非平凡的泛化界。与以往基于Rademacher复杂度的方法不同，作者利用PAC-Bayes理论证明了只要目标函数的稀疏度不超过上下文长度，就能实现良好泛化。实验和机械可解释性研究支持了理论构造在真实Transformer中的可行性。这项工作为理解Transformer为何能高效泛化提供了新的理论工具。
A Sharper Picture of Generalization in Transformers
We study transformers' generalization behavior on boolean domains from the perspective of the Fourier Spectra of their target functions. In contrast to prior work (Edelman et al., 2022; Trauger and Tewari, 2024), which derived generalization bounds from Rademacher complexity, we investigate the feasibility of obtaining generalization bounds via PAC-Bayes theory. We show that sparse spectra concentrated on low-degree components enable low-sharpness constructions with good generalization properties. Our idea is to show the existence of flat minima implementing any boolean function of sparsity no greater than the context length, and then apply a PAC-Bayes bound to an idealized low-sharpness learner, resulting in a non-vacuous generalization bound. We evaluate predictions empirically and conduct a mechanistic interpretability study to support the realism of our theoretical construction in real transformers.