Research map / Architectures and efficiency / Pruning and efficient tuning
SlimGPT: Layer-wise Structured Pruning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SlimGPT, a layer-wise structured pruning method for LLMs that extends Optimal Brain Surgeon with batched greedy pruning (grouped Cholesky) and a logarithmic Incremental Pruning Ratio schedule, evaluated on LLaMA-family models.
Problem
Structured pruning of LLMs under limited compute is hard: performance degrades strongly at high pruning ratios and post-training recovery is expensive; OBS-style layer-wise methods were not previously applied to structured pruning of LLMs.
Approach
Extends OBS to column/head-level structured pruning with grouped Cholesky decomposition for fast head-wise error estimation, dynamic group size for FFN channel pruning, and a logarithmically increasing per-layer pruning ratio to counter error accumulation in layer-wise pruning.
Key results
- On LLaMA-7B pruned 20%, SlimGPT achieves WikiText2 PPL 16.68 vs 16.80 for LoraPrune and zero-shot average 65.07 vs 61.50 for LLM-Pruner.
- At 50% pruning of LLaMA-7B, SlimGPT with LoRA reaches average 53.76 vs 48.35 for LLM-Pruner; without tuning it reaches 52.23.
- Pruning LLaMA-7B by 20% takes about 678 s on a single A100 with about 7 GB GPU memory per the runtime table; 50% pruning reduces inference memory to about 51% of the original.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add logarithmically increasing layer-wise pruning ratios instead of uniform ratios (ablation) | SlimGPT · LLaMA-7B pruned 50%, without post-training | PPL: 123.05 -> 38.83 | Helped |
| Add logarithmically increasing layer-wise pruning ratios instead of uniform ratios (ablation) | SlimGPT · LLaMA-7B pruned 50%, without post-training | zero-shot average: 44.34 -> 52.23 | Helped |
| Use logarithmic rather than linear increasing pruning ratios (ablation) | SlimGPT · LLaMA-7B pruned 50%, without post-training | PPL: 46.57 -> 38.83 | Helped |
| Use logarithmic rather than linear increasing pruning ratios (ablation) | SlimGPT · LLaMA-7B pruned 50%, without post-training | zero-shot average: 53.45 -> 52.23 | Hurt |
| Add Dynamic Group Size instead of fixing FFN group size at 128 (ablation) | SlimGPT · LLaMA-7B pruned 50%, without post-training | zero-shot average: 51.63 -> 52.23 | Helped |
| Add grouped Cholesky decomposition for attention head error estimation (ablation) | SlimGPT · LLaMA-7B pruned 50%, without post-training | PPL: 54.94 -> 38.83 | Helped |
| Add LoRA post-training (ablation) | SlimGPT · LLaMA-7B pruned 50% | zero-shot average: 52.23 -> 53.76 | Helped |
| Add LoRA post-training (ablation) | SlimGPT · LLaMA-7B pruned 50% | BoolQ | Hurt |
| Use C4 rather than Alpaca calibration data (ablation) | SlimGPT · LLaMA-7B pruned 50%, without post-training | PPL: 48.26 -> 42.06 | Helped |
| Use Alpaca rather than C4 calibration data (ablation) | SlimGPT · LLaMA-7B pruned 50%, without post-training | zero-shot average: 54.44 -> 52.70 | Hurt |
| Prune with SlimGPT rather than LoRAPrune | LLaMA-7B · WikiText2 validation, 20% pruning | PPL: 16.80 -> 16.68 | Helped |
| Prune with SlimGPT rather than LLM-Pruner | LLaMA-7B · Commonsense Reasoning zero-shot average, 20% pruning | average score: 61.50 -> 65.07 | Helped |
Surprising findings
- Without fine-tuning, SlimGPT on Vicuna-7B gave higher WikiText2 PPL than LLM-Pruner at 20% pruning, which the authors attribute to calibration data mismatch with an instruction-tuned model.
- On MMLU, LoRA fine-tuning slightly decreased SlimGPT's LLaMA2-7B 20%-pruned performance (39.4 w/o tune vs 37.8 with tune).
Limitations
- Performance degradation remains significant at high pruning ratios (e.g., 50%) and on complex tasks such as LongBench.
- The logarithmic Incremental Pruning Ratio is a naive choice and not claimed to be optimal.
- Calibration on pretraining data is less suited to instruction-tuned models such as Vicuna, where SlimGPT without tuning underperforms on PPL.
Experimental setup
Datasets: C4 Alpaca WikiText2 BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA MMLU LongBench
Benchmarks: WikiText2 BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA MMLU LongBench
Metrics: perplexity zero-shot accuracy average accuracy latency memory usage
Baselines: LLM-Pruner Compresso LoraPrune
Builds on: Optimal Brain Surgeon OBC GPTQ SparseGPT LLM-Pruner
Closest papers by meaning
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.95)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.93)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.93)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.93)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.93)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.