Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse training from scratch
Trains sparse networks with dynamic or random sparsity instead of pruning dense models.
95 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Pruning and efficient tuning, keyword cluster “sparsity · compression · structured”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
30 papers in 2023–24 and 24 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 10.8% to 4.3%.
No longer reported by papers here: ImageNet (top-1 accuracy, last in 2024); CIFAR-10 (accuracy, last in 2023).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · DSD: Dense-Sparse-Dense Training for Deep Neural Networks (ICLR 2017 · 159 citations)
- 2018 · Deep Rewiring: Training very sparse deep networks (ICLR 2018 · 162 citations)
- 2019 · Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization (ICML 2019 · 119 citations)
- 2020 · Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression (CVPR 2020 · 212 citations)
- 2021 · Dynamic Slimmable Network (CVPR 2021 · 146 citations)
- 2022 · The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training (ICLR 2022 · 34 citations)
- 2023 · Structured Sparsity Learning for Efficient Video Super-Resolution (CVPR 2023 · 33 citations)
- 2024 · TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge (ICML 2024 · 8 citations)
- 2025 · Dynamic Sparse Training of Diagonally Sparse Networks (ICML 2025 · 1 citation)
- 2026 · SlideSparse: Fast and Flexible (2N-2):2N Structured Sparsity (ICML 2026 · 0 citations)
Earliest papers here
- DSD: Dense-Sparse-Dense Training for Deep Neural Networks (ICLR 2017 · 159 citations)
- Exploring Sparsity in Recurrent Neural Networks (ICLR 2017 · 138 citations)
- Structured Bayesian Pruning via Log-Normal Multiplicative Noise (NeurIPS 2017 · 131 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ImageNet · top-1 accuracy (higher is better)
6 papers, 2019–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 73.3 | Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization (ICML 2019 · 119 citations) |
| 2022 | 80.1 | Learning Best Combination for Efficient N:M Sparsity (NeurIPS 2022) |
CIFAR-10 · accuracy (higher is better)
4 papers, 2018–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 93.4 | NestedNet: Learning Nested Sparse Structures in Deep Neural Networks (CVPR 2018 · 54 citations) |
| 2020 | 95.05 | Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers (ICLR 2020 · 27 citations) |
What has been tried
73 papers here report 874 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 9%, hurt 19%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add TinyTrain's few-shot meta-training to pre-training | Meta-Dataset nine-dataset average · top-1 accuracy | +5.6 pp on average | Helped | TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge (ICML 2024 · 8 citations) |
| Learn layer sparsity rates instead of keeping the ERK-initialized rates fixed | Multiple sparsity rates; especially RegNetX-400M at 80% sparsity · accuracy | about 10% improvement | Helped | Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes (AAAI 2024 · 7 citations) |
| use the full-dataset learning-rate schedule without faster decay | CIFAR10, 10% · accuracy | 83.9\% -> 91.7\% | Hurt | Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning (ICLR 2024 · 7 citations) |
| Add neuron ablation to constant-fan-in SRigL | CIFAR-10, 99% sparsity · test accuracy | 91.5± 0.3 -> 92.8± 0.1 | Helped | Dynamic Sparse Training with Structured Sparsity (ICLR 2024 · 4 citations) |
| add adaptive warmup perturbations instead of using Taylor pruning with SAM | ImageNet-C, size 0.20 · Top1 accuracy | 34.49 -> 35.86 | Helped | Adaptive Sharpness-Aware Pruning for Robust Sparse Networks (ICLR 2024 · 3 citations) |
| Use uniform rather than greedy soup merging | CIFAR-100, 98% target sparsity, Phase 3, m=10 · accuracy | 72.44 -> 73.05 | Helped | Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging (ICLR 2024 · 3 citations) |
| add BCSR conversion of diagonal matrices for the GPU kernel (vs direct diagonal computation) | ImageNet-1K · training time (hrs) | 18.07 -> 11.42 hrs | Helped | Dynamic Sparse Training of Diagonally Sparse Networks (ICML 2025 · 1 citation) |
| Use a linear sparsity schedule instead of searched layer sparsities | ImageNet ResNet50 Trojan patches · Average AUC | 95.78 -> 93.51 | Hurt | SPADE: Sparsity-Guided Debugging for Deep Neural Networks (ICML 2024 · 1 citation) |
| remove weight rescaling | ImageNet-1K · top-1 accuracy | 78.8 -> 78.5 | Hurt | S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training (NeurIPS 2024 · 1 citation) |
| Add re-training after BLAST16 compression | Common-sense reasoning benchmarks · Average zero-shot accuracy | Helped | BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference (NeurIPS 2024 · 1 citation) | |
| Add the projected-point gradient during x-minimization | ResNet-20, CIFAR-10, 99.5% sparsity · validation accuracy | 79.55 (± 0.13) -> 85.85 (± 0.09 ) | Helped | SAFE: Finding Sparse and Flat Minima to Improve Pruning (ICML 2025 · 0 citations) |
| Increase sparsity in large networks | DMC Hard · episode return | Helped | Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning (ICML 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 15 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 5 of 15 papers
- CIFAR-100 · 5 of 15 papers
- ImageNet-1K · 4 of 15 papers
- C4 · 3 of 15 papers
- WikiText2 · 2 of 15 papers
- ImageNet · 2 of 15 papers
Baselines
- SparseGPT · 3 of 15 papers
- Wanda · 3 of 15 papers
- ALPS · 2 of 15 papers
- magnitude pruning · 2 of 15 papers
- RigL · 2 of 15 papers
- AC/DC · 2 of 15 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 54 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 4 papers
- Google · 4 papers
- Cerebras Systems · 4 papers
- University of Texas, Austin · 3 papers
- University of Texas at Austin · 3 papers
- Zhejiang University · 2 papers
- University of Aberdeen · 2 papers
- Clemson University · 2 papers
- University of Oxford · 2 papers
- Carnegie Mellon University · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Structured network pruning (162 papers)
- Joint pruning and quantization (78 papers)
- Layer-wise pruning of LLMs (91 papers)
Elsewhere in the topic
- Low-rank compression and merging (226 papers)
- Adapter-based parameter-efficient tuning (149 papers)
- LoRA variants and theory (121 papers)
- Parameter-efficient transfer analysis (97 papers)
- Efficient ViT adaptation (62 papers)
- Lottery ticket discovery (42 papers)
- Pruning at initialization (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).