Statistical Framework Unifies Adam, Shampoo and Muon Optimizers
A Statistical Viewpoint on Adam, Shampoo and Muon
The practical value lies in the unification: one statistical principle explains Adam, Shampoo and Muon, so practitioners can tune or combine these optimizers with a clearer theoretical map instead of treating them as folklore. Engineers training GPT-scale models get concrete alternatives, since MND beats Distributed Shampoo and KL-Shampoo on stability and runtime, and MND-Muon wins 21 of 24 paired runs against Muon. Researchers in optimizer design can extend the covariance-approximation framework to new preconditioners.
A new arXiv paper (2610.24839) models gradients as a normal distribution and argues that stochastic gradients should be whitened using their covariance. It derives a family of methods via regularized likelihood estimation, where diagonal approximations recover a method closely related to Adam and Kronecker approximations yield the new MN method, a variant of Shampoo and KL-Shampoo. Applying the framework to gradient direction recovers Muon, and adding whitening produces MN-Muon, a new preconditioned variant. In experiments on GPT-2-type models at Chinchilla scale, the diagonal implementation MND outperforms Distributed Shampoo and KL-Shampoo on stability, final validation loss and runtime, while MND-Muon improves on Muon in 21 of 24 paired GPT-small runs. NorMuon remains comparable, achieving a slightly lower validation loss on GPT-medium and at GPT-small batch 512 on FineWeb.
A Statistical Viewpoint on Adam, Shampoo and Muon
We provide new statistical insights into designing stochastic optimization methods. By modeling the gradient via a normal distribution, we argue that the stochastic gradient should be whitened using its covariance. We give an online method for estimating the mean and variance of the gradients based on regularized likelihood or, equivalently, through a maximum a posteriori estimate. We then show how to use structured approximations to the covariance within this same framework, which includes diagonal and Kronecker-factored approximations. The diagonal approximation results in a method that is very closely related to Adam and performs as well in practice. The Kronecker approximation results in the new MN method, which is a variant of Shampoo and KL-Shampoo. We also recover Muon by applying this same statistical framework to the gradient direction rather than the full gradient. By combining this with a whitening step, we arrive at MN-Muon, a novel preconditioned version of Muon. Our experiments on GPT-2-type models at Chinchilla scale indicate that MND, the diagonal implementation of MN, outperforms both the Distributed Shampoo and the KL-Shampoo implementations in terms of stability, final validation loss and time taken. MND-Muon, the diagonal implementation of MN-Muon, improves on Muon in 21 of 24 paired GPT-small runs and achieves a lower best validation loss on GPT-medium, where it remains stable over the six tested learning rates. However, we found that NorMuon was comparable to MND-Muon and achieved a slightly lower validation loss on GPT-medium and at GPT-small batch 512 on FineWeb.
