Research map / Architectures and efficiency / Pruning and efficient tuning
SlimLLM: Accurate Structured Pruning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SlimLLM, a structured pruning method for LLMs that scores attention heads by Pearson similarity and FFN channels by PCA-based feature-space importance, recovers accuracy with per-dimension linear regression, and sets per-layer pruning ratios via cosine similarity.
Problem
Structured pruning of LLMs requires accurately determining sub-module (head/channel) importance and minimizing performance loss, while choosing per-layer pruning ratios; existing methods aggregate element-wise weight importance and use uniform ratios.
Approach
Evaluate heads and channels as whole units rather than aggregating element importance: heads by Pearson similarity between original and head-removed outputs (with greedy search), FFN channels by PCA eigenvalue-weighted feature-space importance, recover performance with per-dimension least-squares linear regression on output matrices, and allocate layer pruning ratios from input-output cosine simil
Key results
- At 20% pruning on LLaMA-7B without tuning, average commonsense score 61.22 vs 60.53 for LoRAP (Llama-7B unpruned 63.25).
- At 50% pruning on LLaMA-7B without tuning, average 50.10 vs 47.25 for LoRAP; with tuning 53.16 vs 52.17.
- Decoding latency on a single V100 drops 28.5% at 50% pruning (13.12s to 9.38s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SlimLLM instead of LoRAP at 20% pruning without tuning | LLaMA-7B · Commonsense Reasoning average | Average: 60.53 -> 61.22 | Helped |
| Use SlimLLM instead of LoRAP at 20% pruning with tuning | LLaMA-7B · Commonsense Reasoning average | Average: 61.7 -> 62.41 | Helped |
| Use SlimLLM instead of LoRAP at 50% pruning without tuning | LLaMA-7B · Commonsense Reasoning average | Average: 47.25 -> 50.10 | Helped |
| Use SlimLLM instead of LoRAP at 50% pruning with tuning | LLaMA-7B · Commonsense Reasoning average | Average: 52.17 -> 53.16 | Helped |
| Use SlimLLM instead of LoRAP at 20% pruning without tuning | LLaMA2-7B · Commonsense Reasoning average | Average: 59.44 -> 60.44 | Helped |
| Use SlimLLM instead of LoRAP at 20% pruning with tuning | LLaMA2-7B · Commonsense Reasoning average | Average: 61.24 -> 61.70 | Helped |
| Use SlimLLM instead of LoRAP at 50% pruning without tuning | LLaMA2-7B · Commonsense Reasoning average | Average: 46.09 -> 46.93 | Helped |
| Use SlimLLM instead of LoRAP at 50% pruning with tuning | LLaMA2-7B · Commonsense Reasoning average | Average: 52.31 -> 52.02 | Hurt |
| Add greedy head search (ablation) | SlimLLM · LLaMA-7B, 50% pruning | Average: 48.35 -> 50.10 | Helped |
| Add feature space importance (ablation) | SlimLLM · LLaMA-7B, 50% pruning | Average: 49.74 -> 50.10 | Helped |
| Use non-uniform instead of uniform layer pruning (ablation) | SlimLLM · LLaMA-7B, 50% pruning | Average: 42.16 -> 50.10 | Helped |
| Set layer-ratio alpha to 7 rather than 1, 4, or 10 (ablation) | SlimLLM · LLaMA-7B, 50% pruning | Average: 44.42; 44.95; 50.10; 46.86 | Mixed |
Surprising findings
- On LLaMA2-7B with tuning at 50% pruning, LoRAP slightly outperforms SlimLLM on average (52.31 vs 52.02), which the authors attribute to finetuning strategy mismatch.
Limitations
- Linear regression only applicable when output dimensions match, so restricted to MHA and FFN output matrices.
- Greedy head search not applied to FFN channels due to computational overhead.
- Layer-ratio hyperparameter alpha must be tuned per pruning ratio.
Experimental setup
Datasets: BookCorpus Alpaca (cleaned)
Benchmarks: WikiText2 PTB BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA
Metrics: perplexity zero-shot accuracy average accuracy decoding latency
Baselines: LLM-Pruner LoRAPrune LoRAP
Builds on: LLM-Pruner LoRAP LoRAPrune OWL
Closest papers by meaning
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.95)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.94)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.94)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.93)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.93)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.