Research map / Architectures and efficiency / Mixture of experts
SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
Qifan Yu, Xinyu Ma, Zhijian Zhuo, Minrui Wang, Deyi Liu, Shiyi Zhan, Yiyuan Ma, liang xiang, Xingyan Bin, Di He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RMS-preserving rescaling at expansion (ablation) | OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · MoE Inner 2x, all init pairs | late-stage reference loss | Improved |
| add RMS-preserving rescaling at expansion (ablation) | OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Hidden 2x | late-stage reference loss | Improved |
| copy both up and down projections (copy-copy) vs other RMS-preserved inits (ablation) | OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · MoE Inner 2x | late-stage loss | Hurt |
| copy vs random for RMSNorm weight expansion (ablation) | OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · hidden expansion | final loss | No clear effect |
| drop optimizer state for all parameters (symmetric) (ablation) | OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Inner 2x copy-copy | post-expansion recovery and final loss | Hurt |
| copy optimizer states to new parameters (symmetric) (ablation) | OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Inner 2x copy-copy | post-expansion recovery and final loss | Hurt |
| asymmetric optimizer state reset (reset only new channels) (ablation) | OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Inner 2x copy-copy | recovery speed and final loss | Improved |
| scale optimizer states along with parameters on top of asymmetric reset (ablation) | OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Inner 2x copy-copy | final loss | No clear effect |
| add asymmetric learning rate re-warmup (rho 1.3, 250 steps) (ablation) | OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Inner 2x, Hidden 2x, joint | final loss | Improved |
| use SPARKLING instead of naive function-preserving copy scaled | OLMoE MoE · Hidden 2x & Inner 2x | downstream average: 60.89 -> 62.55 | Improved |
| use SPARKLING instead of from-scratch expanded model | OLMoE MoE · Inner 2x | downstream average: 59.64 -> 60.30 | Improved |
| use SPARKLING instead of from-scratch expanded model | OLMoE MoE · Hidden 2x | downstream average: 60.46 -> 61.13 | Improved |
Closest papers by meaning
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.84)
- Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks (NeurIPS 2026 · similarity 0.84)
- A Stable Whitening Optimizer for Efficient Neural Network Training (NeurIPS 2025 · similarity 0.83)
- GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.83)
- Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.83)
- Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
