AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Controlled LLM Training on Spectral Sphere

Tian Xie, Haoming Luo, Haoyu Tang, Hu Yiwen, Jason Klein Liu, Qingnan Ren, Yang Wang, Xin Zhao, Rui Yan, Bing Su, Chong Luo, Baining Guo

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
constrain weights to the spectral sphere in addition to Muon's update constraint (SSO vs Muon)Dense 1.7B pretraining · commonsense benchmarks averageaverage accuracy: 55.26 -> 56.35Improved
use SSO instead of AdamW for dense pretrainingDense 1.7B · validation losstraining speedup: 1.24x faster than AdamWImproved
use exact steepest-descent tangent solver (Lagrange multiplier) vs MuonSphere with lambda=0 (ablation)Spectral Sphere · Dense 1.7B commonsense averageaverage accuracy: 56.19 -> 56.35Improved
use spectral mu-P learning rate scaler instead of Align-Adam-RMS scaler (ablation)Spectral Sphere · Dense validation loss (LR grid)validation lossImproved
split fused QKV per head for initialization and optimization (ablation)Spectral Sphere · Dense training losstrain lossImproved
split FFN gate and up matrices (ablation)Spectral Sphere · Dense training losstrain lossNo clear effect
set spectral radius scale c to a moderate value (c = 2.0) versus other values (ablation)Spectral Sphere · Dense final loss across LRfinal lossImproved
use SSO with spectral normalization for MoE expertsMoE 8B-A1B · router load balancingMaxVioImproved
use SSO for 200-layer DeepNetDeepNet 200 layers · training lossLM lossImproved
remove weight decay on hidden 2D weights under spectral retraction (ablation)Spectral Sphere · Dense 1.7BlossNo clear effect
use SSO instead of Muon for end-to-end step timeDense 1.7B on B200 · per-step latencytime (ms): 6878.83 -> 7666.32 (+11.45%)Hurt
add Triton SYRK kernel applied to all matrix sizes (ablation)SSO implementation · per-step latencytime (ms): 9365.5 -> 10284.2Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy