Research map / Large language models / Language, safety and interpretability
A Simple and Effective Pruning Approach for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Wanda, a training-free one-shot LLM pruning method that scores each weight by its magnitude times the input activation norm and prunes per output row, without any weight update.
Problem
Pruning large language models without retraining or expensive second-order weight reconstruction, since magnitude pruning fails badly on LLMs and SparseGPT requires costly weight updates.
Approach
Score each weight as |W_ij| times the L2 norm of the corresponding input feature over calibration data, and compare weights per output row rather than per layer; no weight update is applied, so the pruned model is used as is.
Key results
- At 50% unstructured sparsity, Wanda reaches WikiText perplexity 7.26 on LLaMA-7B versus 17.29 for magnitude pruning and 7.22 for SparseGPT.
- Wanda matches dense LLaMA-65B and LLaMA-2-70B zero-shot accuracy at 50% unstructured sparsity (66.67% vs 66.97% dense for LLaMA-65B).
- Wanda's pruning metric computation is reported as up to 300x faster than SparseGPT, and it remains robust with as few as one calibration sample (perplexity 7.66).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Wanda instead of magnitude pruning | LLaMA-7B · WikiText validation, unstructured 50% | perplexity: 17.29 -> 7.26 | Helped |
| Use Wanda instead of SparseGPT | LLaMA-7B · WikiText validation, unstructured 50% | perplexity: 7.22 -> 7.26 | No clear effect |
| Use Wanda instead of magnitude pruning | LLaMA-7B · 7 zero-shot tasks, unstructured 50% | mean accuracy: 46.94 -> 54.21 | Helped |
| Use Wanda instead of SparseGPT | LLaMA-7B · 7 zero-shot tasks, unstructured 50% | mean accuracy: 54.94 -> 54.21 | Hurt |
| Compare weights per-output rather than layer-wise (ablation) | Wanda on LLaMA-7B · WikiText validation, unstructured 50% | perplexity: 7.95 -> 7.26 | Helped |
| Use the Wanda weight-and-activation metric instead of magnitude with per-output comparison (ablation) | LLaMA-7B · WikiText validation, unstructured 50% | perplexity: 13.41 -> 7.26 | Helped |
| Add sequential weight update (ablation) | Wanda on LLaMA-7B · WikiText validation, unstructured 50% | perplexity: 7.26 -> 7.32 | No clear effect |
| Add sequential weight update (ablation) | Wanda on LLaMA-7B · WikiText validation, 2:4 | perplexity: 11.53 -> 10.89 | Helped |
| Reduce calibration samples from 128 to 1 (ablation) | Wanda on LLaMA-7B · WikiText validation, unstructured 50% | perplexity: 7.26 -> 7.46 | No clear effect |
| Increase unstructured sparsity from 50% to 80% (ablation) | Wanda on LLaMA-7B · WikiText validation | perplexity: 7.26 -> 5e3 | Hurt |
Surprising findings
- Pruning per output row beats per-layer pruning for LLMs, yet the same comparison group does not help image classifiers (per-layer was slightly better there).
- Weight update (as in SparseGPT) gives no improvement to Wanda at 50% and 4:8 sparsity, suggesting LLM sparse subnetworks exist without weight modification.
Limitations
- Unstructured sparsity above 50% without retraining degrades substantially (e.g., 80% perplexity in the thousands for LLaMA-7B), and the paper notes a significant gap to dense models.
- Speedup results are simulated kernel latency rather than end-to-end system deployment; 2:4 structured sparsity gives 1.24x end-to-end on LLaMA-7B.
- Benefit of per-output grouping was not observed for image classifiers, so the finding may be specific to LLMs.
Experimental setup
Datasets: C4 WikiText BoolQ RTE HellaSwag WinoGrande ARC-e ARC-c OpenbookQA MMLU ImageNet-1K
Benchmarks: WikiText perplexity BoolQ RTE HellaSwag WinoGrande ARC-e ARC-c OpenbookQA MMLU
Metrics: perplexity mean zero-shot accuracy 5-shot accuracy pruning time inference latency
Baselines: Magnitude pruning SparseGPT SNIP BERT-LTH Movement pruning Platon PINS
Builds on: Optimal Brain Surgeon Optimal Brain Damage SparseGPT magnitude pruning
Closest papers by meaning
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.93)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.92)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.92)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.92)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.