这篇论文把Adam的底裤扒干净了，还顺手做了个Adam-mini，显存省一半，效果不掉。搞训练优化的人值得看看。
论文为Adam优化器建立理论基础，揭示其收敛性受问题相关的相变支配：在恰当选择、依赖批量大小的超参数下Adam收敛，而小β₂下可能发散。通过分析Transformer训练的Hessian结构，发现其趋向近块对角形态且块间异质性强，这解释了Adam为何优于SGD。作者基于随机矩阵理论严格证明该结构来源，并由此设计出Adam-mini，将优化器内存占用减少50%且保持性能。结果对Muon等其他优化器的理解与改进也有启示。
On the Principles Behind Neural Network Optimizers
Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation. This thesis develops a principled grounding for Adam and motivates new designs. First, we revisit Adam's divergence--convergence debate and show the existence of a problem-dependent phase transition: with properly chosen, batch-size-dependent hyperparameters, Adam converges, whereas under small-$β_2$ regimes it can diverge. Second, we investigate why Adam substantially outperforms SGD on Transformers through Hessian structure. We find that the Hessian evolves toward a near-block-diagonal form along training, accompanied by strong block heterogeneity. We prove that this structure makes Adam's diagonal preconditioner effective. We further show that this special Hessian structure originates from consecutive multiplications of large matrix variables, and we provide a rigorous analysis based on random matrix theory. Finally, these insights motivate Adam-mini, a new optimizer that reduces Adam's memory footprint by 50\% while preserving its performance. Our results also have broader implications beyond Adam: they reveal new local structures in matrix-based nonconvex problems, and also help understand and improve recent NN optimizers, such as Muon.