Survey of Optimizers
Blog: Survey of Optimizers
The survey provides practical guidance for selecting optimizers based on specific training conditions, helping developers optimize performance for different model scales and computational constraints.
This paper surveys neural network optimization methods from 2025-2026, organizing them along four axes: temporal estimation, update geometry, horizon management, and representation systems. It examines optimizers like Muon, Shampoo, and SOAP, comparing matrix-aware methods with AdamW. The study finds no context-independent replacement for AdamW, as optimizer performance varies with model scale, batch size, and evaluation metrics.
Blog: Survey of Optimizers
Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical conclusion is deliberately non-triumphal: matrix-aware methods represent a genuine advance, but there is no context-independent replacement for AdamW. Rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. The practical consequence is a compositional view of optimizer design and a stricter protocol for evaluating optimizer claims.
