Research map / Architectures and efficiency / RNNs and normalization
Controlled LLM Training on Spectral Sphere
Tian Xie, Haoming Luo, Haoyu Tang, Hu Yiwen, Jason Klein Liu, Qingnan Ren, Yang Wang, Xin Zhao, Rui Yan, Bing Su, Chong Luo, Baining Guo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| constrain weights to the spectral sphere in addition to Muon's update constraint (SSO vs Muon) | Dense 1.7B pretraining · commonsense benchmarks average | average accuracy: 55.26 -> 56.35 | Improved |
| use SSO instead of AdamW for dense pretraining | Dense 1.7B · validation loss | training speedup: 1.24x faster than AdamW | Improved |
| use exact steepest-descent tangent solver (Lagrange multiplier) vs MuonSphere with lambda=0 (ablation) | Spectral Sphere · Dense 1.7B commonsense average | average accuracy: 56.19 -> 56.35 | Improved |
| use spectral mu-P learning rate scaler instead of Align-Adam-RMS scaler (ablation) | Spectral Sphere · Dense validation loss (LR grid) | validation loss | Improved |
| split fused QKV per head for initialization and optimization (ablation) | Spectral Sphere · Dense training loss | train loss | Improved |
| split FFN gate and up matrices (ablation) | Spectral Sphere · Dense training loss | train loss | No clear effect |
| set spectral radius scale c to a moderate value (c = 2.0) versus other values (ablation) | Spectral Sphere · Dense final loss across LR | final loss | Improved |
| use SSO with spectral normalization for MoE experts | MoE 8B-A1B · router load balancing | MaxVio | Improved |
| use SSO for 200-layer DeepNet | DeepNet 200 layers · training loss | LM loss | Improved |
| remove weight decay on hidden 2D weights under spectral retraction (ablation) | Spectral Sphere · Dense 1.7B | loss | No clear effect |
| use SSO instead of Muon for end-to-end step time | Dense 1.7B on B200 · per-step latency | time (ms): 6878.83 -> 7666.32 (+11.45%) | Hurt |
| add Triton SYRK kernel applied to all matrix sizes (ablation) | SSO implementation · per-step latency | time (ms): 9365.5 -> 10284.2 | Hurt |
Closest papers by meaning
- From SGD to Muon: Adaptive Optimization via Schatten-p Norms (NeurIPS 2026 · similarity 0.87)
- u-$\mu$P: The Unit-Scaled Maximal Update Parametrization (ICLR 2025 · similarity 0.86)
- Tensorion: A Tensor-Aware Generalization of the Muon Optimizer (NeurIPS 2026 · similarity 0.85)
- Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds (NeurIPS 2026 · similarity 0.85)
- Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory (NeurIPS 2026 · similarity 0.85)
- Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
