Research map / Architectures and efficiency / Pruning and efficient tuning
The LLM Surgeon
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLM Surgeon, a Kronecker-factored curvature (K-FAC-style) pruning framework that computes global-threshold masks and correlated weight updates for unstructured, semi-structured, and structured pruning of pretrained LLMs (OPT, Llama-v2) without retraining from scratch.
Problem
Pruning large pretrained language models is hard to do well: existing post-training methods ignore gradient information and mostly handle only unstructured or semi-structured sparsity, and none gave practically usable structured (row/column) pruning of LLMs.
Approach
Scales Kronecker-factored Fisher approximations (G ⊗ A) to LLMs to derive OBS-like removal costs for elements, rows and columns, uses a global threshold for dynamic sparsity allocation, applies correlated closed-form weight updates, prunes in multiple shots with curvature re-estimation, and optionally interleaves low-rank (LoRA) first-order corrections.
Key results
- Structured pruning of OPT-6.7b and Llama-v2-7B to 80% size keeps WikiText-2 PPL near baseline (Llama-v2-7B 7.29 vs 5.12 dense at 80% with G⊗A correlations; 5.25 vs 5.12 at 90% with full correlations).
- Unstructured 2:4-equivalent semi-structured 50% pruning gives OPT-125m PPL 44.64 vs SparseGPT 45.51 and LLM Surgeon outperforms baselines on OPT-1.3b/2.7b/6.7b at 50%.
- On Llama-v2 13B, structured pruning to 80% gives PPL 5.286 vs K-OBD 6.294 (dense 4.547), and the authors report under 2% downstream drop at 20% compression.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use full correlations between rows and columns instead of correlations limited within rows/columns (ablation) | LLM Surgeon structured pruning · WikiText-2, OPT and Llama-v2 models | PPL: Llama-v2 (7b), 90%: 5.43 -> 5.25; 50%: 25.62 -> 15.38 | Helped |
| Use full correlations between rows and columns instead of correlations limited within rows/columns (ablation) | LLM Surgeon structured pruning · WikiText-2, OPT models | PPL: OPT (125m), 90%: 28.29 -> 28.01; OPT (2.7b), 90%: 12.00 -> 12.02 | Mixed |
| Add interleaved LoRA updates between pruning shots (ablation) | LLM Surgeon structured pruning on OPT-125m · WikiText-2, 50% target size | PPL: 49.78 -> 36.50 | Helped |
| Add interleaved LoRA updates between pruning shots (ablation) | LLM Surgeon structured pruning · WikiText-2, larger models | PPL | No clear effect |
| Use LLM Surgeon instead of K-OBD for structured pruning | OPT and Llama-v2 models · WikiText-2, structured compression | PPL: Llama-v2 (7b), 50%: 46.64 -> 15.38; OPT (6.7b), 90%: 10.53 -> 10.77 | Mixed |
| Use LLM Surgeon instead of SparseGPT | OPT models · WikiText-2, semi-structured 2:4 | PPL: OPT (125m): 45.51 -> 44.64; OPT (1.3b): 29.44 -> 25.10; OPT (2.7b): 14.92 -> 14.64; OPT (6.7b): 13.01 -> 12.10 | Helped |
| Use LLM Surgeon instead of SparseGPT | OPT and Llama-v2 models · WikiText-2, unstructured 50% target | PPL: OPT (125m): 33.17 -> 30.30; OPT (1.3b): 26.77 -> 15.47; OPT (2.7b): 12.88 -> 12.68; OPT (6.7b): 11.92 -> 10.97; Llama-v2 | Helped |
| Use LLM Surgeon instead of K-OBD | Llama-v2-13B · Structured pruning, 90% target size | PPL: 4.908 -> 4.692 | Helped |
| Use LLM Surgeon instead of K-OBD | Llama-v2-13B · Structured pruning, 50% target size | PPL: 16.06 -> 9.428 | Helped |
Surprising findings
- Interleaved LoRA first-order corrections help the smallest OPT-125m model but not the larger models, which the authors hypothesise is overfitting on the small calibration set.
- Structured compression to 50% is not better than simply training a smaller model of similar size, per the authors.
Limitations
- Structured pruning degrades noticeably beyond about 20-30% compression, especially on downstream tasks like LAMBADA.
- Calibration uses only 128 sequences of WikiText-2, and performance may depend on calibration data; larger models may overfit to small calibration sets.
- Relies on a local quadratic (Gaussian/Laplace) approximation that becomes unreliable for large weight jumps, motivating multi-shot pruning.
Experimental setup
Datasets: WikiText-2 C4 French Wikipedia German Wikipedia Italian Wikipedia
Benchmarks: WikiText-2 perplexity wikitext word perplexity BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenBookQA COPA LAMBADA WSC273
Metrics: test perplexity zero-shot accuracy
Baselines: Magnitude pruning L-OBD K-OBD SparseGPT
Builds on: Optimal Brain Surgeon Optimal Brain Damage Eigendamage KFAC Optimal Brain Compression
Closest papers by meaning
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.92)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.90)
- You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning (ICLR 2025 · similarity 0.90)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.90)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.90)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.