Research map / Large language models / Language, safety and interpretability
The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically searches sparse pre-training schedules for LLaMA-2-style models and shows that a Chinchilla-form scaling law using average parameter count over pre-training unifies sparse and dense loss prediction.
Problem
How should iterative magnitude pruning be scheduled during LLM pre-training, and how do scaling laws change when parameter count decreases over training (sparse pre-training) versus fixed-size dense pre-training?
Approach
Replace the parameter-count term N in the Chinchilla law with the average active parameter count over pre-training; a theoretical argument (log loss linear in log compute, compute proportional to active parameters) plus experiments show this unifies sparse and dense loss prediction.
Key results
- Pruning starting at 25% of total compute and ending at 75% achieves near-optimal final evaluation loss across training durations and sparsity levels.
- The fitted average-parameter scaling law predicts final loss with average absolute error of 0.016 across 30 configurations.
- A 1.14B model at 50% sparsity reaches perplexity 2.44 versus 2.46 for the parameter-matched dense model; downstream geometric mean accuracy 42.42% vs 42.45%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Match sparse and dense models on average parameter count and training tokens | Sparse pre-training versus dense pre-training · 1B-scale validation | perplexity: 2.44 vs 2.46 | No clear effect |
| Match sparse and dense models on average parameter count | Sparse pre-training versus dense pre-training · PIQA | accuracy: 69.7 vs 69.26 | No clear effect |
| Match sparse and dense models on average parameter count | Sparse pre-training versus dense pre-training · ARC Easy | accuracy: 42.05 vs 41.58 | No clear effect |
| Match sparse and dense models on average parameter count | Sparse pre-training versus dense pre-training · ARC Challenge | accuracy: 24.74 vs 26.37 | No clear effect |
| Match sparse and dense models on average parameter count | Sparse pre-training versus dense pre-training · Lambada | accuracy: 36.52 vs 34.62 | No clear effect |
| Match sparse and dense models on average parameter count | Sparse pre-training versus dense pre-training · Winogrande | accuracy: 51.85 vs 52.41 | No clear effect |
| Allocate 25% of total compute to dense training and 50% to weight removal (ablation) | Sparse pre-training · 162M-10x and 162M-20x schedule sweep | evaluation loss: within 0.01 of the minimum | Helped |
| Use the optimal dense-pretraining learning rate and batch size (ablation) | Sparse pre-training · 162M models | evaluation loss: within 0.01 difference | Helped |
| Allocate 25% rather than 0% or 50% of compute to dense training (ablation) | Sparse pre-training · 20%–80% sparsity; 10x and 20x training regimes | evaluation loss: 7 out of 8 cases | Helped |
| Increase dense training allocation to 50% (ablation) | Sparse pre-training · High-sparsity models | evaluation loss | Hurt |
| Allocate 25% of compute to iterative pruning for 10x training, versus 50% for 20x training (ablation) | Sparse pre-training · 162M-10x and 162M-20x | evaluation loss: 2.83; 2.77 | Helped |
| Extend iterative pruning allocation beyond 50% (ablation) | Sparse pre-training · High-sparsity models (80%) | evaluation loss | Hurt |
Surprising findings
- Sparse pre-training with matched average parameter count matches dense final loss, implying a lossless 2x compression at 80% final sparsity.
- Under a lifetime-compute analysis, sparse pre-training recommends training with less data than Beyond Chinchilla prescriptions (31.4% lifetime compute saving in the example).
Limitations
- Computational savings not demonstrated because unstructured sparsity lacks hardware/software support; actual compute may be up to 1/(1-sparsity) times the effective estimate.
- Scaling-law fit error is larger at 60% sparsity (0.03), attributed to unmodeled regularization effects of sparsity.
- Scale limited to models up to 468M in the main fits; the 1B+ validation uses a single run.
Experimental setup
Datasets: C4 (en)
Benchmarks: PIQA ARC Easy ARC Challenge Lambada Winogrande
Metrics: final evaluation loss perplexity accuracy geometric mean accuracy mean absolute prediction error
Baselines: Chinchilla scaling law Beyond Chinchilla dense pre-training sparse scaling law (Frantar et al.)
Builds on: Chinchilla scaling law iterative magnitude pruning Lottery Ticket Hypothesis Beyond Chinchilla scaling
Closest papers by meaning
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.92)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.92)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.92)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.92)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.92)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.