Research map / Architectures and efficiency / Pruning and efficient tuning
Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares state-of-the-art pruning algorithms against an exhaustive-style combinatorial search for the sparsest MLP on a synthetic Cubist Spiral dataset, finding pruning consistently produces much denser networks than the search finds.
Problem
Whether current pruning algorithms can recover the sparsest possible network that achieves a target accuracy is unknown, because real datasets and models make the optimal sparsity hard to assess.
Approach
Builds a synthetic Cubist Spiral dataset and a two-phase combinatorial search (structured neuron-count grid search, then unstructured mask enumeration) to find near-minimal sparse 4-layer MLPs, then benchmarks nine pruning methods against these benchmarks, including under optimal width and initialization.
Key results
- Combinatorial search reaches above 95% accuracy with 30 nonzeros, while the best pruning method (LTH) needs 44 nonzeros.
- For above 99.5% accuracy, the search finds a 45-nonzero model, while the next best (GMP) needs 61 nonzeros.
- Out of 18,954 pruning experiments under optimal width, only two matched or beat the combinatorial search (SNIP at 29 nonzeros with 95.14%, GMP at 40 nonzeros with 99.68%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use combinatorial search instead of pruning to find sparse masks | 4-layer MLP on Cubist Spiral · Cubist Spiral, 95%-99.5% accuracy regime | nonzero parameters to reach accuracy threshold: 44 -> 30 | Helped |
| use combinatorial search instead of pruning to find sparse masks | 4-layer MLP on Cubist Spiral · Cubist Spiral, >99.5% accuracy | nonzero parameters to reach accuracy threshold: 61 -> 45 | Helped |
| reduce network width to 3 (ablation) | pruning algorithms on 4-layer MLP · Cubist Spiral | accuracy | Helped |
| increase network width, including to 256 (ablation) | pruning algorithms on 4-layer MLP · Cubist Spiral | accuracy | Hurt |
| provide pruning algorithms with the combinatorial search's optimal width and initialization (ablation) | pruning algorithms · Cubist Spiral | accuracy | Hurt |
| use Bench-995 initialization for combinatorial search (ablation) | combinatorial search · Cubist Spiral, >99.5% accuracy | nonzero parameters: 45 -> 38 | Helped |
| prune Bench-995 after training rather than search masks (ablation) | Bench-995 · Cubist Spiral, >99.5% accuracy | nonzero parameters: 38 -> 42 | Hurt |
| use Bench-95 initialization for combinatorial search (ablation) | combinatorial search · Cubist Spiral, approximately 95% accuracy | nonzero parameters: 30 -> 26 | Helped |
| increase width under random-mask pruning (ablation) | RigL and other pruning methods · Cubist Spiral | disconnected paths | Hurt |
| magnitude-prune the trained sparse model further (ablation) | pruned models · Cubist Spiral | accuracy | No clear effect |
Surprising findings
- Overparameterization hurts most pruning methods; at width 256 nearly all pruning algorithms largely fail.
- Pruning after training is insufficient: magnitude pruning the best model gives 42 nonzeros while a fresh search with the same initialization finds 38.
Limitations
- Restricted to synthetic data and small 4-layer MLPs of width 16 due to combinatorial search cost.
- Combinatorial search gives only a lower bound on achievable sparsity, with no guarantee of the optimal sparsest model.
- Search over masks was restricted to subsets for the 99.5% target due to compute limits.
Experimental setup
Datasets: Cubist Spiral
Benchmarks: Cubist Spiral
Metrics: accuracy number of nonzero parameters FLOPS
Baselines: GMP LTH GraSP SNIP SynFlow Iter SNIP FORCE ProsPr RigL
Builds on: Lottery Ticket Hypothesis SynFlow RigL GMP
Closest papers by meaning
- How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective (NeurIPS 2024 · similarity 0.91)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.91)
- Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging (ICLR 2024 · similarity 0.90)
- SAFE: Finding Sparse and Flat Minima to Improve Pruning (ICML 2025 · similarity 0.90)
- No Free Prune: Information-Theoretic Barriers to Pruning at Initialization (ICML 2024 · similarity 0.89)
- GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.