Research map / Architectures and efficiency / Pruning and efficient tuning
Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DSA, an evolutionary expression-discovery framework that searches for layer-wise sparsity allocation functions and plugs them into uniform LLM pruning methods (Wanda, SparseGPT, magnitude) to improve accuracy.
Problem
Uniform layer-wise sparsity in LLM pruning ignores varying layer importance and degrades performance at high sparsity; existing non-uniform allocation methods are either costly (BESA) or hand-designed with fixed hyperparameters (OWL).
Approach
Automatically search, via evolutionary algorithm over a computation-graph search space of pre-process, reduction, transform, and post-process operators, for a function mapping element-wise importance scores to per-layer sparsity ratios, then apply it to existing uniform pruning metrics.
Key results
- At 50% sparsity, DSA with SparseGPT improves LLaMA-3 8B zero-shot mean accuracy by 10.65% (51.77 to 62.41) and LLaMA-1 7B by 4.73%.
- On WikiText-2 at 65%-80% sparsity for LLaMA-1-7B with Wanda, DSA reaches perplexity 12.62 at 65% versus OWL 13.05, and 736.81 at 80% versus OWL 1002.87.
- On LLaVA-1.5 Vicuna-7B at 50% sparsity, DSA achieves 76.08 VQAv2, 65.57 SQA, and 54.36 VQA, the best among compared pruning methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DSA layer-wise sparsity allocation with Wanda pruning | GSM8K, LLaMA-1/2 and Mistral models at 50% sparsity · GSM8K | accuracy: 8.19 -> 8.22; 15.60 -> 15.64; 8.11 -> 8.47; 13.42 -> 14.27; 25.40 -> 25.78 | Helped |
| Apply DSA layer-wise sparsity allocation with Wanda pruning | MMLU, LLaMA-1/2 and Mistral models at 50% sparsity · MMLU | accuracy: 29.81 -> 31.05; 38.29 -> 39.76; 33.06 -> 33.08; 48.06 -> 48.38; 53.05 -> 53.87 | Helped |
| Apply DSA layer-wise sparsity allocation with Wanda pruning | LLaVA-1.5 Vicuna-7B, 50% sparsity · VQAv2, SQA, VQA | score: 75.86 -> 76.08; 63.99 -> 65.57; 53.69 -> 54.36 | Helped |
| Apply DSA layer-wise sparsity allocation with Wanda pruning | LLaVA-1.5 Vicuna-13B, 50% sparsity · VQAv2, SQA, VQA | score: 78.62 -> 79.10; 71.19 -> 73.17; 58.23 -> 58.70 | Helped |
| Replace OWL layer-wise allocation with DSA allocation | Wanda-pruned LLaMA-1-7B · WikiText-2, 65% sparsity | perplexity: 13.05 -> 12.62 | Helped |
| Replace OWL layer-wise allocation with DSA allocation | Wanda-pruned LLaMA-1-7B · WikiText-2, 80% sparsity | perplexity: 1002.87 -> 736.81 | Helped |
| Use evolutionary search instead of random search | DSA allocation-function search for LLaMA-1 7B · WikiText-2 search | perplexity | Helped |
Surprising findings
- Gains from the searched allocation function depend strongly on the transform operation; for example, the discovered function uses cosine and exponential transforms rather than an intuitive monotone mapping.
Limitations
- The search itself incurs nontrivial cost, as acknowledged in the limitations section.
- The search is performed once on LLaMA-1-7B WikiText-2 perplexity and transferred to other models and tasks, so transferability relies on that single search.
- The stability analysis is local and does not provide a formal theory of why the discovered functions work.
Experimental setup
Datasets: WikiText-2 C4
Benchmarks: Winogrande OpenBookQA HellaSwag BoolQ ARC RTE GSM8K MMLU VQAv2 SQA VQA WikiText-2
Metrics: accuracy perplexity
Baselines: Magnitude SparseGPT Wanda OWL BESA ECoFLaP Global ER ER-plus Uniform
Builds on: Wanda SparseGPT OWL BESA
Closest papers by meaning
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.93)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.93)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.92)
- BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation (ICLR 2024 · similarity 0.92)
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.92)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.