Research map / Architectures and efficiency / Pruning and efficient tuning
Layer-wise pruning of LLMs
Allocates sparsity across layers for one-shot or structured pruning of large language models.
91 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2020–2026) · an approach in Pruning and efficient tuning, keyword cluster “sparsity · compression · structured”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
19 papers in 2023–24 and 70 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 6.8% to 12.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2020 · Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (ICML 2020 · 105 citations)
- 2022 · From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression (AAAI 2022 · 19 citations)
- 2023 · SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot (ICML 2023 · 68 citations)
- 2024 · Resource-Efficient Transformer Pruning for Finetuning of Large Models (CVPR 2024 · 14 citations)
- 2025 · Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · 10 citations)
- 2026 · SepPrune: Structured Pruning for Efficient Deep Speech Separation (AAAI 2026 · 11 citations)
Earliest papers here
- Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (ICML 2020 · 105 citations)
- From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression (AAAI 2022 · 19 citations)
- SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot (ICML 2023 · 68 citations)
What has been tried
50 papers here report 565 measured changes (a component added, removed or swapped, compared with a baseline): helped 62%, no clear effect 9%, hurt 24%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Increase the pruning ratio to 66.6% | CIFAR100 · accuracy | 88.34 -> 72.90 | Hurt | Resource-Efficient Transformer Pruning for Finetuning of Large Models (CVPR 2024 · 14 citations) |
| Add logarithmically increasing layer-wise pruning ratios instead of uniform ratios | LLaMA-7B pruned 50%, without post-training · PPL | 123.05 -> 38.83 | Helped | SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · 12 citations) |
| add knowledge distillation during sparse retraining | WikiText-2, GPT2/OPT · perplexity | 40.34 -> 39.29; 29.79 -> 29.08; 28.65 -> 27.21; 39.46 -> 36.97 | Helped | Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · 10 citations) |
| use FP32 instead of double precision for PCA | WikiText-2 perplexity, Llama 70B at 25% slicing · perplexity | 4.60 -> 7.01 | Hurt | SliceGPT: Compress Large Language Models by Deleting Rows and Columns (ICLR 2024 · 8 citations) |
| Use the simple deepest-layer heuristic instead of similarity-informed block selection | MMLU · accuracy | Mixed | The Unreasonable Ineffectiveness of the Deeper Layers (ICLR 2025 · 5 citations) | |
| Train for more epochs rather than the 1-epoch default | WikiText2 · perplexity | 6.86 -> 6.85 (3 epochs); 6.86 -> 6.84 (10 epochs); 6.86 -> 6.86 (30 epochs) | No clear effect | BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation (ICLR 2024 · 3 citations) |
| Increase per-token magnitude pruning of both caches from 50% to 70% sparsity | LongBench · average score | 42.65 -> 40.96 | Hurt | MUSTAFAR: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference (NeurIPS 2025 · 1 citation) |
| add PCG weight refinement after magnitude pruning | MP-determined support · relative reconstruction error | 1.98e-1 -> 8.63e-2 | Helped | ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models (NeurIPS 2024 · 1 citation) |
| Place the dense or low-rank projection before the block-diagonal projection | RefinedWeb, 2.2B training tokens · PPL | 29.17 -> 29.65 | Hurt | Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · 1 citation) |
| replace fixed-threshold pruning with TopK+GS | WikiText2 test · perplexity | 6.80 -> 6.02 | Helped | La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · 0 citations) |
| Add greedy head search | LLaMA-7B, 50% pruning · Average | 48.35 -> 50.10 | Helped | SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · 0 citations) |
| Add online error-accumulation-minimization reconstruction to SVD-LLM pruning | WikiText2, 50% density, LLaMA2-7B · perplexity | 33.27 -> 16.55 | Helped | Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models (ICML 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 40 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- WikiText-2 · 18 of 40 papers
- C4 · 14 of 40 papers
- HellaSwag · 8 of 40 papers
- WinoGrande · 8 of 40 papers
- Alpaca · 7 of 40 papers
- BoolQ · 7 of 40 papers
- ARC-Easy · 7 of 40 papers
- ARC-Challenge · 7 of 40 papers
Baselines
- Wanda · 11 of 40 papers
- SparseGPT · 10 of 40 papers
- ShortGPT · 8 of 40 papers
- SliceGPT · 8 of 40 papers
- LLM-Pruner · 7 of 40 papers
- Magnitude · 4 of 40 papers
- LaCo · 4 of 40 papers
- SLEB · 4 of 40 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 89 papers; from OpenAlex and the papers' affiliations).
- Google · 5 papers
- Hong Kong University of Science and Technology · 5 papers
- Tsinghua University · 4 papers
- Meituan · 4 papers
- IBM · 3 papers
- Alibaba Group · 3 papers
- Advanced Micro Devices · 3 papers
- National University of Singapore · 3 papers
- Shanghai Jiao Tong University · 3 papers
- Harbin Institute of Technology · 3 papers
Use it
Download BibTeX of 28 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Structured network pruning (162 papers)
- Sparse training from scratch (95 papers)
- Joint pruning and quantization (78 papers)
Elsewhere in the topic
- Low-rank compression and merging (226 papers)
- Adapter-based parameter-efficient tuning (149 papers)
- LoRA variants and theory (121 papers)
- Parameter-efficient transfer analysis (97 papers)
- Efficient ViT adaptation (62 papers)
- Lottery ticket discovery (42 papers)
- Pruning at initialization (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).