Research map / Architectures and efficiency / Pruning and efficient tuning
SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SparseGPT instead of magnitude pruning at 50% sparsity | OPT-175B · raw-WikiText2 | perplexity: 4.3e4 -> 8.21 | Helped |
| Use SparseGPT instead of magnitude pruning at 50% sparsity | OPT-175B · PTB | perplexity: 2.3e3 -> 12.37 | Helped |
| Use SparseGPT instead of magnitude pruning at 50% sparsity | OPT-175B · C4 subset | perplexity: 1.7e3 -> 10.36 | Helped |
| Increase structured pruning from 4:8 to 2:4 | SparseGPT on OPT-175B · raw-WikiText2 | perplexity: 8.45 -> 8.74 | Hurt |
| Jointly apply 50% sparsity and 4-bit quantization instead of GPTQ 3-bit quantization | OPT-175B · raw-WikiText2 | perplexity: 8.68 vs. 8.29 | Helped |
| Prune the first half of layers to 2:4 rather than all layers (ablation) | OPT-175B · raw-WikiText2 | perplexity: 8.74 -> 8.22 | Helped |
| Prune the first half of layers to 2:4 rather than all layers (ablation) | OPT-175B · PTB | perplexity: 13.25 -> 12.15 | Helped |
| Prune the first half of layers to 2:4 rather than all layers (ablation) | OPT-175B · C4 subset | perplexity: 10.92 -> 10.22 | Helped |
| Increase calibration data sample count (ablation) | SparseGPT on OPT-2.7B · raw-WikiText2 | perplexity | Helped |
| Set Hessian dampening very high (ablation) | SparseGPT on OPT-2.7B · raw-WikiText2 | perplexity | Hurt |
| Use column-wise or near-full mask-selection blocks instead of reasonable-sized blocks (ablation) | SparseGPT on OPT-2.7B · raw-WikiText2 | perplexity | Hurt |
| Vary the random calibration-data sampling seed (ablation) | SparseGPT on OPT-2.7B · raw-WikiText2 | perplexity: 13.52 ± 0.075 | No clear effect |
Closest papers by meaning
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.90)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.90)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.89)
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns (ICLR 2024 · similarity 0.89)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.89)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.