Research map / Architectures and efficiency / Pruning and efficient tuning
Scaling Laws for Sparsely-Connected Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a joint scaling law L(S,N,D) for weight-sparse Transformers (ViT/JFT-4B and T5/C4), uses it to characterize optimal sparsity and sparse-vs-dense equivalence, and extends it to n:m sparsity and pruning from pretrained checkpoints.
Problem
How weight sparsity affects the scaling behavior of Transformer foundation models trained on massive datasets, where dense models keep improving with longer training, so it is unclear whether sparse models can win under fair compute comparisons.
Approach
A joint scaling law in which sparsity enters as a multiplicative saturating power-law factor on the non-zero parameter term, with the data term kept independent of sparsity; this yields analytic optimal-sparsity contours parallel to the Chinchilla line and dense-equivalent size gains.
Key results
- The fitted joint law L(S,N,D) matches T5/C4 and ViT/JFT sweeps, and extrapolates to a 2.3B-parameter model pruned to 75% sparsity.
- A 75% sparse ViT with N non-zeros matches a dense model with about 2.17N parameters, and T5 gives about 2.16x, at the same data.
- Starting from a pretrained dense checkpoint is over 4x more data-efficient than sparsifying from scratch at 50% and 75% sparsity when pretraining cost is excluded.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase unstructured sparsity to 0.750 | ViT and T5 foundation models · JFT and C4 | equivalent dense size multiplier: 2.17×; 2.16× | Helped |
| Use higher unstructured sparsity, from 0.500 to 0.875 | ViT and T5 foundation models · JFT and C4 | equivalent dense size multiplier: 1.60× -> 2.63×; 1.59× -> 2.63× | Helped |
| Train longer than Chinchilla compute optimality and use nonzero sparsity | ViT and T5 scaling-law models · JFT and C4 | validation loss | Helped |
| Use n:m structured patterns instead of unstructured sparsity | T5 · C4 | equivalent dense size multiplier: 1.56×; 1.62×; 1.67×; 1.81× | Mixed |
| Prune a pretrained checkpoint rather than sparsifying from scratch when its pretraining cost is already sunk | ViT S/16, M/16, and B/16 · JFT | data required for equivalent performance: 4.90×; 4.27×; 4.35× | Helped |
| Include the pretrained model's pretraining cost when comparing pruning with sparsifying throughout training | ViT S/16, M/16, and B/16 · JFT | relative data required: 0.25×; 0.23×; 0.22×; 0.21×; 0.14×; 0.13× | Mixed |
| Change pruning frequency from 100 to 50 or 200 (ablation) | GMP on ViT-B/16 · 900M images, sparsity 0.9375 | accuracy: 45.28 -> 45.08; 45.28 -> 45.13 | No clear effect |
| Extend the pruning interval from 0.250–0.750 to 0.125–0.875 (ablation) | GMP on ViT-B/16 · 900M images, sparsity 0.9375 | accuracy: 45.28 -> 45.33 | No clear effect |
| Shorten the pruning interval to 0.475–0.625 (ablation) | GMP on ViT-B/16 · 900M images, sparsity 0.9375 | accuracy: 45.28 -> 44.65 | Hurt |
| Change pruning interval to 0.125–0.625 (ablation) | GMP on ViT-B/16 · 900M images, sparsity 0.9375 | accuracy: 45.28 -> 45.13 | No clear effect |
| Change pruning interval to 0.375–0.875 (ablation) | GMP on ViT-B/16 · 900M images, sparsity 0.9375 | accuracy: 45.28 -> 45.04 | No clear effect |
| Replace GMP with AC/DC (ablation) | T5 · C4, 250K steps, 0.500 sparsity | perplexity: 17.11 -> 17.11; 6.11 -> 6.11 | No clear effect |
Surprising findings
- Optimal sparsity increases with training length; sparse models overtake dense ones when training well beyond Chinchilla-optimal, with 50% sparsity optimal for under 2x (ViT) and under 3x (T5) longer training in sparse FLOPs.
- Gains from 75% sparsity are similar across vision and text (about 2.15x), despite different domains.
Limitations
- The sparsification recipe (gradual magnitude pruning) was optimized for robustness across scales, not for maximal pruning performance, so coefficients may be improvable.
- Analysis targets non-zero parameter count and assumes sparsity is perfectly exploited in training, not realized hardware speedups.
- Studied only the infinite-data, pretraining-loss regime; sparsity may behave differently under limited or repeated data.
Experimental setup
Datasets: JFT-4B C4
Benchmarks: JFT-4B validation loss C4 validation loss
Metrics: validation loss Huber loss of fit dense-equivalent size multiplier relative data required
Baselines: dense Transformers at equal parameter count AC/DC GMP
Builds on: Chinchilla scaling laws gradual magnitude pruning Jaxpruner
Closest papers by meaning
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.91)
- Asymmetric Scaling Laws from Sparse Features (NeurIPS 2026 · similarity 0.90)
- Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data (NeurIPS 2024 · similarity 0.89)
- Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets (NeurIPS 2025 · similarity 0.89)
- Unified Scaling Laws for Compressed Representations (NeurIPS 2025 · similarity 0.88)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.