Research map / Large language models / Language, safety and interpretability
SparseLLM: Towards Global Pruning of Pre-trained Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SparseLLM, a global-pruning framework that decomposes LLM pruning into coordinated subproblems via auxiliary variables and alternating optimization, improving over SparseGPT and Wanda especially at high sparsity.
Problem
Global pruning of LLMs is memory-prohibitive, while local layer-wise pruning (SparseGPT, Wanda) yields suboptimal models, especially at high sparsity.
Approach
Reformulate the LLM as a chain of modules with auxiliary variables (layer outputs and activations) and soft penalty constraints, so FFN modules are pruned globally via alternating closed-form subproblem updates, while MHA layers use a local solver like SparseGPT.
Key results
- At 80-90% unstructured sparsity on OPT models, SparseLLM reduces perplexity substantially versus SparseGPT (e.g., OPT-30b WT2 at 80%: 15.61 vs 16.49; OPT-1.3b at 90%: 1350 vs 2597 WT2), with up to ~80% reduction claimed at high sparsity.
- At 2:4 semi-structured sparsity, SparseLLM matches or slightly beats SparseGPT (e.g., OPT-6.7b WT2 11.07 vs 11.30).
- Computation time per layer is substantially higher than SparseGPT (e.g., OPT-66b 284.59s vs 103.19s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SparseLLM instead of SparseGPT for 90% pruning | OPT-1.3b · WT2, 90% sparsity | perplexity: 2596.70 -> 1350.31 | Helped |
| Use SparseLLM instead of SparseGPT for 70% pruning | OPT-2.7b · WT2, 70% sparsity | perplexity: 13.79 -> 13.82 | Hurt |
| Use SparseLLM instead of SparseGPT for 80% pruning | OPT-1.3b · WT2, 80% sparsity | perplexity: 69.67 -> 58.92 | Helped |
| Use SparseLLM instead of SparseGPT for 70% pruning | LLaMA-2 7b · WT2, 70% sparsity | perplexity: 15.98 -> 16.15 | Hurt |
| Use SparseLLM instead of SparseGPT for 90% pruning | LLaMA-2 7b · WT2, 90% sparsity | perplexity: 344.97 -> 225.23 | Helped |
| Use SparseLLM instead of SparseGPT for 70% pruning | LLaMA-3 8b · WT2, 70% sparsity | perplexity: 22.37 -> 20.98 | Helped |
| Use SparseLLM instead of SparseGPT for 2:4 pruning | OPT-1.3b · WT2, 2:4 sparsity | perplexity: 15.11 -> 14.97 | Helped |
| Use SparseLLM instead of SparseGPT at 90% sparsity | OPT-13b · BoolQ, 90% sparsity | accuracy: 47.49 -> 53.43 | Helped |
| Use SparseLLM instead of SparseGPT at 90% sparsity | LLaMA-2 7b · Mean zero-shot tasks, 90% sparsity | accuracy: 31.99 -> 32.92 | Helped |
| Vary alpha and beta; alpha=0.1, beta=0.1 gives the lowest reported perplexity (ablation) | SparseLLM on OPT-1.3b · PTB, 70% sparsity | perplexity: 18.04 -> 17.82 | Mixed |
Surprising findings
- On LLaMA-2 models, improvements over SparseGPT are smaller than on OPT, and on some PTB results SparseLLM is slightly worse.
Limitations
- Assumes homogeneous sparsity across layers (same sparsity per layer as global sparsity).
- Only prunes the first 50% of decoder layers in experiments for compute reasons.
- Substantially higher pruning computation time than SparseGPT.
Experimental setup
Datasets: C4
Benchmarks: WikiText2 PTB C4 BoolQ RTE HellaSwag WinoGrande ARC-e ARC-c OBQA
Metrics: perplexity zero-shot accuracy
Baselines: Magnitude pruning SparseGPT Wanda
Builds on: SparseGPT Wanda AdaPrune ADMM
Closest papers by meaning
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.93)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.93)
- MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (NeurIPS 2024 · similarity 0.93)
- ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models (NeurIPS 2024 · similarity 0.92)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.92)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.