Muon Optimizer Outperforms Adam in LLM Training
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
SAMuon reduces training token requirements by 13.3-24.0% compared to Muon, offering significant efficiency gains for researchers and companies training large language models in ASEAN.
Researchers analyzed Muon, an orthogonal optimizer that accelerates large language model pretraining compared to Adam and SGD. Through spectral probing of Transformer loss landscapes, they found Muon's anisotropic spectral profile allows larger step sizes in the tolerant bulk than the volatile head operating at the Edge-of-Stability. This explains why Muon outperforms Adam and SGD. The team then developed Spectral-Aware Muon (SAMuon), which improves Muon by amplifying the bulk using a static spectral prior.
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
Orthogonal optimisers such as Muon can substantially accelerate large language model pretraining relative to Adam, yet the mechanism remains incompletely understood. We investigate this through an out-of-sample spectral probing analysis of Transformer loss landscapes. At checkpoints along real training trajectories, we decompose each momentum buffer into its singular directions and estimate the loss-optimal step size along each direction on held-out data. The resulting spectral profile is anisotropic yet stable across batches and training stages, and consistent across the optimisers and model scales: a volatile head operating at the Edge-of-Stability supports a much smaller step size than the tolerant bulk, which permits substantially larger steps. This profile provides a unified spectral allocation account of why Muon outperforms Adam, which outperforms SGD. It also exposes a limitation of Muon's uniform scaling: it still underutilises the bulk. Guided by this finding, we introduce Spectral-Aware Muon (SAMuon), which holds the head at the Muon scale and amplifies the bulk using a static spectral prior. We provide two variants: the complete SAMuon follows the measured profile using a low-rank randomised SVD and the simplified SAMuon-lite uses a two-level approximation via rank-one power iteration. Neither method adds persistent optimiser state or notable extra FLOPs beyond Muon at scale, and the idealised exact-whitening versions of both retain Muon's asymptotic convergence rate under standard assumptions. Across "modded-nanogpt" models from 124M to 1B parameters, both variants outperform tuned AdamW and Muon (Scion implementation) baselines in all evaluated model-scale and batch-size configurations. SAMuon requires 13.3% to 24.0% fewer training tokens to reach the same validation loss as Muon, while SAMuon-lite retains most of this gain with near-zero wall-clock overhead.
