Research map / Large language models / Language, safety and interpretability
Compressing LLMs: The Truth is Rarely Pure and Never Simple
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LLM-KICK, a knowledge-intensive benchmark suite that re-evaluates training-free pruning and quantization of Vicuna models beyond perplexity, finding that pruning degrades knowledge-intensive performance at modest sparsity while retrieval-augmented and summarization abilities hold up better.
Problem
Perplexity is an insufficient metric for judging whether training-free, data-free compression (pruning and quantization) preserves the capabilities of LLMs, since compressed models stay close to perplexity of dense models while losing knowledge-intensive abilities.
Approach
Assemble a curated multi-task benchmark (factoid QA, MMLU, in-context retrieval-augmented QA, in-context summarization, MT-Bench-style instruction following) and define "matching" compressed LLMs as those within 5% of dense performance, to evaluate SoTA pruning (Magnitude, SparseGPT, Wanda) and quantization (GPTQ) methods on Vicuna models.
Key results
- On FreebaseQA, all SoTA pruning methods fall below the 5% matching tolerance at around 30-35% unstructured sparsity, and no matching N:M sparse model was found.
- On MMLU, compressed Vicuna-13B at 46.16% sparsity with Wanda or SparseGPT reaches 45.3% and 46.3% accuracy, versus 46.7% for dense Vicuna-7B, while magnitude pruning reaches only 31.7%.
- Pruned LLMs at 50% sparsity remain matching for open-book (retrieval-augmented) QA, with Vicuna-7B matching up to about 40% sparsity and 8-bit quantization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply unstructured pruning, including magnitude pruning, SparseGPT, or Wanda | Vicuna-7B · FreebaseQA | accuracy | Hurt |
| Use structured N:M pruning | Vicuna models · LLM-KICK knowledge-intensive tasks | task performance | Hurt |
| Use quantization instead of pruning | Vicuna models · LLM-KICK tasks | task performance | Helped |
| Provide relevant external knowledge in the prompt | Compressed Vicuna models · TriviaQA, in-context retrieval augmented QA | task performance | Helped |
| Apply compression to summarize increasingly long stories | Compressed Vicuna-7B · CNN/DailyMail in-context summarization | summary quality | Hurt |
| Apply 50% unstructured magnitude pruning (ablation) | Vicuna-7B · MMLU | accuracy: 0.471 -> 0.050 | Hurt |
| Apply 50% unstructured SparseGPT pruning (ablation) | Vicuna-7B · MMLU | accuracy: 0.471 -> 0.308 | Hurt |
| Apply 50% unstructured Wanda pruning (ablation) | Vicuna-7B · MMLU | accuracy: 0.471 -> 0.386 | Hurt |
| Increase Wanda calibration sample count (ablation) | Vicuna-7B · MMLU at 70% sparsity | zero-shot performance | No clear effect |
| Increase SparseGPT calibration sample count (ablation) | Vicuna-7B · MMLU at 70% sparsity | zero-shot performance | Helped |
| Add 2-3 in-context examples (ablation) | Wanda-pruned Vicuna-7B · MMLU | performance at high sparsity | Helped |
| Use LLM-int8() rather than AWQ or GPTQ 8-bit | Vicuna-7B · Factoid-QA, MMLU, Retrieval Augmented QA, Instruction Following | task performance | Helped |
Surprising findings
- Compressed Vicuna-13B at the same parameter count as dense Vicuna-7B does not outperform the smaller dense model on MMLU with one-shot magnitude pruning.
- Simple one-shot magnitude pruning performs comparably to or better than SoTA pruning methods within the matching regime on MCR-QA and instruction following.
Limitations
- Evaluation restricted mainly to Vicuna (decoder-only) models due to licensing and performance considerations.
- LLM-as-judge evaluation with GPT-4 relies on a proprietary model and a 5% tolerance threshold chosen by the authors.
- Does not test whether lost knowledge can be recovered; recovery via fine-tuning is left to future work.
Experimental setup
Datasets: FreebaseQA TriviaQA MMLU CNN/DailyMail MT-Bench prompts WikiText
Benchmarks: FreebaseQA MMLU TriviaQA CNN/DailyMail MT-Bench
Metrics: accuracy exact match perplexity GPT-4 score (0-10) unique token count
Baselines: Magnitude pruning SparseGPT Wanda GPTQ AWQ LLM.int8()
Builds on: SparseGPT Wanda GPTQ MT-Bench
Closest papers by meaning
- LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression (NeurIPS 2026 · similarity 0.93)
- Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression (ICML 2025 · similarity 0.92)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.92)
- Accuracy is Not All You Need (NeurIPS 2024 · similarity 0.92)
- LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment (NeurIPS 2024 · similarity 0.92)
- Soft Prompt Recovers Compressed LLMs, Transferably (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.