NVIDIA 研究：AdamW 在大批量训练中存在规模上限

New research from NVIDIA. Does AdamW have a scale ceiling? This work claims yes, and shows where it sits. At batch sizes up to 100M tokens for next-token prediction, SOAP and Muon maintain training stability and quality while AdamW degrades. Higher-order optimizers have promised faster convergence for a while. The standing objection has been computational cost and numerical stability at scale. The team identifies instabilities in SOAP at large batch sizes and eliminates the loss spikes with per-step QR orthogonalization and improved preconditioning strategies. They also measure Muon's orthogonalization quality empirically rather than assuming it. On multi-billion-parameter models trained over trillions of tokens, both optimizers consistently beat AdamW. A layer-wise distributed optimizer compatible with Megatron-LM balances memory and hides communication without approximating the optimizer math, so the convergence benefit survives the systems layer. Paper: https://t.co/WFy3YEQms2 Learn to build effective AI agents in our academy: https://t.co/1e8RZKs4uX