Background: Limitations of the Muon Optimizer
Optimizers play a crucial role in training large language models. While Muon has been widely adopted in cutting-edge models such as DeepSeek V4 and GLM-5, recent research uncovered a serious flaw: during early training, Muon causes over 25% of neurons in MLP layers to become permanently inactive (dead neurons), severely hindering model performance and wasting computational resources.
Root Cause: Uneven Updates Break Orthogonality
Dead neurons occur when gradients become zero for certain neurons, preventing weight updates. Muon's update scheme disrupts orthogonality constraints, leading to early-stage neuron death. This issue is particularly pronounced in models like DeepSeek V4 and GLM-5, where training efficiency is significantly compromised.
Aurora: A Precision Fix Delivering 100x Efficiency
Developed by Tilde Research, the Aurora optimizer directly addresses Muon's deficiencies. By ensuring uniform updates and maintaining orthogonality, Aurora eliminates dead neurons entirely. It achieves a 100x improvement in training efficiency while incurring only a 6% increase in computational overhead. No hyperparameter tuning is required to replace Muon, making adoption seamless.
New State-of-the-Art Benchmarks
In standard benchmark evaluations, Aurora consistently sets new state-of-the-art records, demonstrating its effectiveness in improving model performance. The optimizer's ability to boost training speed without sacrificing quality marks a significant advancement for the AI community.
Open-Source Release Empowers Community
Tilde Research has open-sourced both the Aurora optimizer and a 1.1B-parameter pretrained model, allowing researchers and developers to freely access, modify, and integrate these tools. This move is expected to accelerate the development of next-generation efficient training pipelines.