Research map / Architectures and efficiency / RNNs and normalization
Escaping the Subspace Trap: The Role of Optimizer Geometry in Model Width Expansion
Jiabei Chen, Haoyu Wang, Yang Yu, Yao Xu, Liangdong Wang, Guang Liu, Shizhu He, Jun Zhao, Kang Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Muon instead of AdamW | Qwen3-2.38B symmetric expansion · Downstream average | average accuracy: 46.67 -> 47.88 | Improved |
| use Muon instead of AdamW | Qwen3-2.38B diagonal expansion · Downstream average | average accuracy: 45.96 -> 47.15 | Improved |
| use Muon instead of AdamW | SmolLM2-540M symmetric expansion · Downstream average | average accuracy: 33.74 -> 34.65 | Improved |
| use Muon instead of AdamW | SmolLM2-540M diagonal expansion · Downstream average | average accuracy: 34.14 -> 34.36 | No clear effect |
| use Muon instead of AdamW | SmolLM2-540M diagonal expansion · MMLU | accuracy: 26.80 -> 26.54 | Hurt |
| use Muon instead of AdamW | Qwen3-2.38B symmetric expansion · Big Bench Hard | accuracy: 41.05 -> 44.43 | Improved |
| use Muon instead of AdamW | Qwen3-2.38B symmetric expansion · AGIEval-en | accuracy: 26.30 -> 26.27 | No clear effect |
| use Muon instead of AdamW | Qwen3-2.38B · Training from scratch | training tokens saved to match AdamW loss: 23.0% | Improved |
| use Muon instead of AdamW | Qwen3-2.38B symmetric expansion · Training efficiency | training tokens saved to match AdamW loss: 32.7%-36.0% | Improved |
| use Muon instead of AdamW | SmolLM2-540M symmetric expansion · Training efficiency | training tokens saved to match AdamW loss: up to 44.6% | Improved |
| use Muon instead of AdamW | diagonal expansion models · Training efficiency | training tokens saved to match AdamW loss: 25.1% to 36.6% | Improved |
| use Muon instead of AdamW | Qwen3-2.38B expanded · Block similarity layer 24 up_proj | block similarity: 0.880 -> 0.788 | Improved |
Closest papers by meaning
- Subspace Optimization for Large Language Models with Convergence Guarantees (ICML 2025 · similarity 0.86)
- SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training (NeurIPS 2025 · similarity 0.86)
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.85)
- Masked Structural Growth for 2x Faster Language Model Pre-training (ICLR 2024 · similarity 0.85)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.85)
- Prescriptive Scaling Reveals the Evolution of Language Model Capabilities (ICML 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
