AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

NorMuon: Making Muon more efficient and scalable

Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, Tuo Zhao

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add neuron-wise normalization after orthogonalization to MuonMuon · SlimPajama 1.1Befficiency gain over Adam (%): 10.43 -> 21.74Improved
add neuron-wise normalization after orthogonalization to MuonMuon · SlimPajama 5.4Befficiency gain over Adam (%): 6.08 -> 13.91Improved
add neuron-wise normalization to MuonMuon · Modded-NanoGPT FineWeb 124Miterations to reach same validation loss: Muon requires 6% more iterationsImproved
add neuron-wise normalization to MuonMuon · Modded-NanoGPT FineWeb 350Miterations to reach same validation loss: Muon requires 15% more iterationsImproved
use neuron-wise instead of coordinate-wise (Adam) normalization after Muon (ablation)NorMuon · 1.1B pretrainingvalidation lossImproved
use neuron-wise instead of coordinate-wise (Adam) normalization after Muon (ablation)NorMuon · 5.4B overheadoptimizer state memory (GB): 40.56 -> 21.19Improved
apply normalization after orthogonalization rather than before (ablation)NorMuon · 1.1B pretrainingvalidation lossImproved
apply normalization to all matrices rather than m>n only (ablation)NorMuon · 1.1B pretrainingvalidation lossImproved
add neuron-wise adaptivity rather than only plain normalization of Muon update (ablation)Muon with normalization · Modded-NanoGPT 350Mvalidation lossImproved
apply NorMuon to all matrices instead of only m>n ones (ablation)NorMuon · Modded-NanoGPT 350Mvalidation lossImproved
add orthogonalization distribution across devices (ablation)NorMuon · 5.4B overheadoptimizer step time (s): 2.29 -> 0.84Improved
use NorMuon instead of AdamWAdamW · 5.4B overheadtraining step time (s): 28.73 -> 29.57Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy