Research map / Architectures and efficiency / Pruning and efficient tuning
You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PruneNet, a calibration-free structured pruning method that learns a stochastic row-selection policy for LLM FFN up-projections with a KS-distance spectral penalty optimized by REINFORCE.
Problem
Structured pruning of LLMs typically relies on external calibration data and degrades sharply at higher compression ratios; intrinsic methods like SliceGPT can even increase parameter count at low ratios.
Approach
Reformulate pruning as policy learning: a small MLP-based policy samples which FFN1 rows to keep (via a reparametrized multinomial), trained with REINFORCE to minimize the Kolmogorov-Smirnov distance between singular-value spectra of original and compressed weights, using no calibration data and reusable across compression ratios.
Key results
- On LLaMA-2-7B at 30% compression without recovery fine-tuning, PruneNet averages 55.45 vs SliceGPT 51.50 across five commonsense tasks (80% vs 75% of dense).
- PruneNet compresses LLaMA-2-7B in about 916 seconds (~15 min) on a single A100, versus about 29 minutes for SliceGPT.
- PruneNet inference throughput at 30% compression is 20.74 tokens/s for LLaMA-2-7B vs 12.82 for SliceGPT and 11.96 dense.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PruneNet instead of SliceGPT | LLaMA-2-7B · 20% compression, commonsense tasks | average zero-shot accuracy: 58.17 -> 61.67 | Helped |
| Use PruneNet instead of SliceGPT | Phi-2 · 20% compression, commonsense tasks | average zero-shot accuracy: 58.15 -> 66.59 | Helped |
| Use a learned pruning policy instead of random parameter selection (ablation) | PruneNet · LLaMA-2-7B zero-shot tasks | average accuracy: 2% | Helped |
| Use stochastic sampling rather than deterministic topk selection (ablation) | PruneNet · LLaMA-2-7B zero-shot tasks | average accuracy: 4% | Helped |
| Replace the KS reward with the Anderson–Darling distance (ablation) | PruneNet · LLaMA-2-7B at different compression ratios | zero-shot accuracy | No clear effect |
| Learn the policy on FFN2 rather than FFN1 (ablation) | PruneNet · LLaMA-2-7B zero-shot tasks | accuracy: < 1% | No clear effect |
| Add recovery fine-tuning on WikiText2 (ablation) | PruneNet-compressed LLaMA-2-7B · 20% compression, commonsense tasks | average zero-shot accuracy: 61.15 -> 62.34 | Helped |
| Add recovery fine-tuning on WikiText2 (ablation) | PruneNet-compressed Phi-2 · 30% compression, commonsense tasks | average zero-shot accuracy: 61.05 -> 58.34 | Hurt |
| Reuse a policy learned at a different compression ratio (ablation) | PruneNet-compressed LLaMA-2-7B · Different compression ratios, zero-shot commonsense tasks | average accuracy: 0.96% | Hurt |
| Vary compression ratios across layers instead of using a fixed 20% ratio (ablation) | PruneNet-compressed LLaMA-2-7B and Phi-2 · Zero-shot commonsense tasks | average accuracy: 2.15%; 0.24% | Mixed |
| Change the recovery fine-tuning dataset (ablation) | PruneNet-compressed LLaMA-2-7B · 20%-30% compression, zero-shot tasks | average accuracy: 1.3; 0.52 | No clear effect |
| Use different compression ratios in different layers (ablation) | PruneNet-compressed LLaMA-2-7B · Average compression ratio 20%, zero-shot tasks | average accuracy: 61.67 -> 59.52 | Hurt |
Surprising findings
- Recovery fine-tuning on WikiText2 gives only a marginal gain (about 1.5%) for compressed LLaMA-2-7B, and it hurts Phi-2 in several cases.
- SliceGPT's throughput on Phi-2 is lower than the dense model's despite compression.
Limitations
- Only sparsifies weight matrices of FFN layers; self-attention compression is harder and yields larger drops.
- Performance drops are still notable at high compression (e.g., 50% ratio gives only about 60% recovery), and MMLU drops significantly for LLaMA and Phi at high ratios.
- Authors note SliceGPT outperforms PruneNet when fine-tuned on Alpaca.
Experimental setup
Datasets: WikiText2 Penn Treebank Alpaca
Benchmarks: PIQA WinoGrande HellaSwag ARC-e ARC-c MMLU
Metrics: zero-shot accuracy average accuracy performance recovery vs dense throughput (tokens/sec) runtime (sec)
Baselines: SliceGPT LLM-Pruner LaCo ShortGPT SVD-LLM ASVD
Builds on: SliceGPT REINFORCE LoRA
Closest papers by meaning
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.93)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.91)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.91)
- Beware of Calibration Data for Pruning Large Language Models (ICLR 2025 · similarity 0.91)
- Let LLM Tell What to Prune and How Much to Prune (ICML 2025 · similarity 0.90)
- The LLM Surgeon (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.