NVIDIA找到AdamW的短板了，批量干到1亿tokens它就崩，但SOAP和Muon依然稳，还解决了训练崩溃的毛病，训大模型可以看看这个。
NVIDIA新研究指出AdamW优化器在大规模训练中存在规模上限，batch size达到1亿tokens进行下一个token预测时AdamW性能下降，而SOAP和Muon保持训练稳定性和质量。研究团队通过每步QR正交化和改进预条件策略解决了SOAP在大batch size下的loss spikes问题。在数十亿参数、数万亿tokens训练中，两种优化器持续优于AdamW。论文地址arxiv.org/abs/2607.20548。
New research from NVIDIA. Does AdamW have a scale ceiling? This work claims yes, and shows where i...
New research from NVIDIA. Does AdamW have a scale ceiling? This work claims yes, and shows where it sits. At batch sizes up to 100M tokens for next-token prediction, SOAP and Muon maintain training stability and quality while AdamW degrades. Higher-order optimizers have promised faster convergence for a while. The standing objection has been computational cost and numerical stability at scale. The team identifies instabilities in SOAP at large batch sizes and eliminates the loss spikes with per-step QR orthogonalization and improved preconditioning strategies. They also measure Muon's orthogonalization quality empirically rather than assuming it. On multi-billion-parameter models trained over trillions of tokens, both optimizers consistently beat AdamW. A layer-wise distributed optimizer compatible with Megatron-LM balances memory and hides communication without approximating the optimizer math, so the convergence benefit survives the systems layer. Paper: arxiv.org/abs/2607.20548 Learn to build effective AI agents in our academy: academy.dair.ai 💬 5 🔄 5 ❤️ 15 👀 2498 📊 8 ⚡