Research map / Large language models / Language, safety and interpretability
LLM-Pruner: On the Structural Pruning of Large Language Models
Xinyin Ma, Gongfan Fang, Xinchao Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| post-train pruned model with LoRA on Alpaca | LLaMA-7B 20% pruned (Element2) · 7 zero-shot tasks | average accuracy: 56.82 -> 60.07 | Improved |
| use block strategy (Element1) instead of channel strategy (ablation) | LLaMA-7B 20% pruned no tune · 7 zero-shot tasks | average accuracy: 45.38 -> 56.69 | Improved |
| use dependency-aware group pruning instead of ignoring dependencies (ablation) | LLaMA-7B 20% Element1 no tune · 7 zero-shot tasks | average accuracy: 38.32 -> 56.69 | Improved |
| use dependency-aware pruning with tuning (ablation) | LLaMA-7B 20% Element1 · WikiText2 | perplexity: 13307.46 -> 17.58 | Improved |
| use gradient-based element importance instead of L2 magnitude (ablation) | LLaMA-7B 20% no tune · 7 zero-shot tasks | average accuracy: 42.65 -> 56.69 | Improved |
| use gradient-based importance instead of random pruning (ablation) | LLaMA-7B 20% no tune · 7 zero-shot tasks | average accuracy: 52.69 -> 56.69 | Improved |
| second-order Element2 instead of first-order Element1 (ablation) | LLaMA-7B 20% w/ tune · 7 zero-shot tasks | average accuracy: 59.23 -> 60.07 | No clear effect |
| element-level importance for ChatGLM-6B instead of vector-level (ablation) | ChatGLM-6B 10% no tune · 6 zero-shot tasks | average accuracy: 46.16 -> 36.45 | Hurt |
| local pruning instead of global pruning (ablation) | LLaMA-7B 20% Element1 · 7 zero-shot tasks | average accuracy: 56.49 -> 56.69 | Improved |
| recover with 2.59M samples instead of 50k (ablation) | LLaMA-5.4B · 7 zero-shot tasks | average accuracy: 59.23 -> 62.36 | Improved |
| combine pruning with LLM.int8() quantization | LLaMA-5.4B · 7 zero-shot tasks | average accuracy: 62.36 -> 62.30 | No clear effect |
| longer LoRA post-training steps (ablation) | LLaMA-7B 20% Element1 · 7 zero-shot tasks | average accuracy: 59.24 -> 59.68 -> 59.32 | Mixed |
Later papers that compared with this method
3 later accepted papers report LLM-Pruner as a baseline in their results tables; 3 of them report a better number on most of the results they share with it.
- Persona-Pruner: Sculpting Lightweight Models for Role-Playing (ICML 2026 · better on 16 of 16 shared results)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · better on 5 of 5 shared results)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · better on 3 of 3 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.94)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.94)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.93)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.93)
- DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models (NeurIPS 2024 · similarity 0.93)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
