DeepMind学者整理了最新优化器研究，帮你理清矩阵方法与AdamW的优劣，不同规模模型该选哪个。
这篇论文综述了2025-2026年神经网络优化器的最新发展。研究将优化器按时间估计、更新几何结构、范围管理和表示系统四个独立维度组织。论文分析了Muon的光谱归一化、Shampoo和SOAP的历史矩阵统计等方法。研究结论表明矩阵感知方法确实有进步，但没有能普遍替代AdamW的优化器。
Blog: Survey of Optimizers
Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical conclusion is deliberately non-triumphal: matrix-aware methods represent a genuine advance, but there is no context-independent replacement for AdamW. Rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. The practical consequence is a compositional view of optimizer design and a stricter protocol for evaluating optimizer claims.