Research map / Architectures and efficiency / RNNs and normalization
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
Boqian Wu, Qiao Xiao, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Mykola Pechenizkiy, Elena Mocanu, Torsten Hoefler, Decebal Constantin Mocanu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sparsity (S=0.5) to the data-saturation threshold of repeated data | LLaMA-2 on C4, data-constrained scaling law · Fitted data saturation scale R_d* | R_d*(S): 4.4 -> 6.9 | Improved |
| use 50% sparsity instead of dense training at ~1e20 FLOPs with 1.3B unique tokens | LLaMA-2 on C4 · Iso-loss, 1.3B unique tokens | parameter-to-token ratio N/D: 0.19 -> 0.064 | Improved |
| add sparsity so optimal allocation shifts to more epochs and fewer non-zero parameters | LLaMA-2 on C4 · Iso-loss, 1.3B unique tokens, ~1e20 FLOPs | optimal epochs: 8 -> 12 | Improved |
| use compute-optimal sparsity instead of dense optimum at U_d=1.3B | LLaMA-2 on C4 · Dense-equivalent loss, U_d=1.3B | non-zero parameters (B): 2.60 -> 0.68 | Improved |
| use compute-optimal sparsity instead of dense optimum at U_d=13B | LLaMA-2 on C4 · Dense-equivalent loss, U_d=13B | non-zero parameters (B): 27.06 -> 6.92 | Improved |
| use compute-optimal sparsity instead of dense optimum at U_d=130B | LLaMA-2 on C4 · Dense-equivalent loss, U_d=130B | non-zero parameters (B): 319.71 -> 70.71 | Improved |
| increase sparsity beyond moderate levels at a fixed non-zero parameter budget | LLaMA-2 on C4, unique data · Validation loss vs sparsity | validation loss | Mixed |
| increase compute-optimal sparsity as unique data grows from 1.3B to 13B tokens | LLaMA-2 on C4 · Compute-optimal sparsity | optimal sparsity: 0.6 -> 0.7 | Improved |
| transfer the dense learning rate directly to sparse training at S=0.75 (versus dense LR x4) (ablation) | DST Llama-1.92B · 5.2B unique tokens, S=0.75 | perplexity: 17.19 vs 17.24 | Improved |
| scale learning rate by 8 (1/(1-S)) for sparse training at S=0.875 (ablation) | DST Llama-1.92B · 5.2B unique tokens, S=0.875 | perplexity: 18.26 vs 18.31 and 18.32 | Improved |
| replace RigL with SET for sparse regrowth (ablation) | DST scaling experiments · Scaling experiments | performance and cost | No clear effect |
Closest papers by meaning
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.92)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.92)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.90)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.90)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.90)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
