AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

When Data Is Scarce: Scaling Sparse Language Models with Repeated Training

Boqian Wu, Qiao Xiao, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Mykola Pechenizkiy, Elena Mocanu, Torsten Hoefler, Decebal Constantin Mocanu

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add sparsity (S=0.5) to the data-saturation threshold of repeated dataLLaMA-2 on C4, data-constrained scaling law · Fitted data saturation scale R_d*R_d*(S): 4.4 -> 6.9Improved
use 50% sparsity instead of dense training at ~1e20 FLOPs with 1.3B unique tokensLLaMA-2 on C4 · Iso-loss, 1.3B unique tokensparameter-to-token ratio N/D: 0.19 -> 0.064Improved
add sparsity so optimal allocation shifts to more epochs and fewer non-zero parametersLLaMA-2 on C4 · Iso-loss, 1.3B unique tokens, ~1e20 FLOPsoptimal epochs: 8 -> 12Improved
use compute-optimal sparsity instead of dense optimum at U_d=1.3BLLaMA-2 on C4 · Dense-equivalent loss, U_d=1.3Bnon-zero parameters (B): 2.60 -> 0.68Improved
use compute-optimal sparsity instead of dense optimum at U_d=13BLLaMA-2 on C4 · Dense-equivalent loss, U_d=13Bnon-zero parameters (B): 27.06 -> 6.92Improved
use compute-optimal sparsity instead of dense optimum at U_d=130BLLaMA-2 on C4 · Dense-equivalent loss, U_d=130Bnon-zero parameters (B): 319.71 -> 70.71Improved
increase sparsity beyond moderate levels at a fixed non-zero parameter budgetLLaMA-2 on C4, unique data · Validation loss vs sparsityvalidation lossMixed
increase compute-optimal sparsity as unique data grows from 1.3B to 13B tokensLLaMA-2 on C4 · Compute-optimal sparsityoptimal sparsity: 0.6 -> 0.7Improved
transfer the dense learning rate directly to sparse training at S=0.75 (versus dense LR x4) (ablation)DST Llama-1.92B · 5.2B unique tokens, S=0.75perplexity: 17.19 vs 17.24Improved
scale learning rate by 8 (1/(1-S)) for sparse training at S=0.875 (ablation)DST Llama-1.92B · 5.2B unique tokens, S=0.875perplexity: 18.26 vs 18.31 and 18.32Improved
replace RigL with SET for sparse regrowth (ablation)DST scaling experiments · Scaling experimentsperformance and costNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy