AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning

Qifan Yu, Xinyu Ma, Zhijian Zhuo, Minrui Wang, Deyi Liu, Shiyi Zhan, Yiyuan Ma, liang xiang, Xingyan Bin, Di He

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Sparse MoE as regularization and attention · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add RMS-preserving rescaling at expansion (ablation)OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · MoE Inner 2x, all init pairslate-stage reference lossImproved
add RMS-preserving rescaling at expansion (ablation)OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Hidden 2xlate-stage reference lossImproved
copy both up and down projections (copy-copy) vs other RMS-preserved inits (ablation)OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · MoE Inner 2xlate-stage lossHurt
copy vs random for RMSNorm weight expansion (ablation)OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · hidden expansionfinal lossNo clear effect
drop optimizer state for all parameters (symmetric) (ablation)OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Inner 2x copy-copypost-expansion recovery and final lossHurt
copy optimizer states to new parameters (symmetric) (ablation)OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Inner 2x copy-copypost-expansion recovery and final lossHurt
asymmetric optimizer state reset (reset only new channels) (ablation)OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Inner 2x copy-copyrecovery speed and final lossImproved
scale optimizer states along with parameters on top of asymmetric reset (ablation)OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Inner 2x copy-copyfinal lossNo clear effect
add asymmetric learning rate re-warmup (rho 1.3, 250 steps) (ablation)OLMoE 0.5B/2.5B MoE, expansion at 100B of 200B tokens · Inner 2x, Hidden 2x, jointfinal lossImproved
use SPARKLING instead of naive function-preserving copy scaledOLMoE MoE · Hidden 2x & Inner 2xdownstream average: 60.89 -> 62.55Improved
use SPARKLING instead of from-scratch expanded modelOLMoE MoE · Inner 2xdownstream average: 59.64 -> 60.30Improved
use SPARKLING instead of from-scratch expanded modelOLMoE MoE · Hidden 2xdownstream average: 60.46 -> 61.13Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy