Research map / Architectures and efficiency / Pruning and efficient tuning
Restoring Pruned Large Language Models via Lost Component Compensation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RestoreLCC, a plug-and-play restoration method that uses contrastive probing to pick critical attention heads in pruned LLMs and learns magnitudes of lost-information directions (from SVD of dense-minus-pruned head activations) to inject back as a bias, recovering performance without adding inference cost.
Problem
Pruned LLMs lose performance, and standard PEFT methods (LoRA, DoRA, representation engineering) designed for dense models recover pruned models suboptimally because they ignore information lost during pruning.
Approach
Estimate the information lost by pruning in attention head activations via SVD of dense-minus-pruned differences, select heads via contrastive NLI-style probing, and learn scalar magnitudes for the lost directions plus a bias vector that is injected into the pruned heads as a constant bias, so inference cost is unchanged.
Key results
- On LLaMA-7B at 50% unstructured sparsity (Wanda), RestoreLCC reaches 58.83% mean accuracy vs 56.82% for the best baseline LoFiT, and PPL 6.93.
- On LLaMA-7B semi-structured 2:4 at 50%, RestoreLCC gets 55.00% mean accuracy, +2.65 over DoRA (52.35%).
- On LLaMA-7B task-specific unstructured 60% sparsity, RestoreLCC averages 61.77% vs 58.21% for LoFiT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RestoreLCC instead of the strongest listed baseline | Wanda-pruned LLaMA-7B, general recovery · Commonsense reasoning | mean accuracy: 56.82 -> 58.83 (+2.01) | Helped |
| Apply RestoreLCC instead of the strongest listed baseline | SparseGPT-pruned LLaMA-7B, general recovery · Commonsense reasoning | mean accuracy: 52.35 -> 55.00 (+2.65) | Helped |
| Apply RestoreLCC instead of the strongest listed baseline | SlimGPT-pruned LLaMA-7B, general recovery · Commonsense reasoning | mean accuracy: 58.51 -> 59.76 (+1.25) | Helped |
| Apply RestoreLCC instead of the strongest listed baseline | Wanda-pruned LLaMA-13B, general recovery · Commonsense reasoning | mean accuracy: 61.42 -> 62.46 (+1.04) | Helped |
| Apply RestoreLCC instead of the strongest listed baseline | SparseGPT-pruned LLaMA-13B, general recovery · Commonsense reasoning | mean accuracy: 55.94 -> 57.67 (+1.73) | Helped |
| Apply RestoreLCC instead of the strongest listed baseline | SlimGPT-pruned LLaMA-13B, general recovery · Commonsense reasoning | mean accuracy: 61.96 -> 63.41 (+1.45) | Helped |
| Apply RestoreLCC instead of the strongest listed baseline | Wanda-pruned LLaMA-7B, task-specific recovery · Commonsense reasoning | mean accuracy: 58.21 -> 61.77 (+3.56) | Helped |
| Apply RestoreLCC instead of the strongest listed baseline | SparseGPT-pruned LLaMA-7B, task-specific recovery · Commonsense reasoning | mean accuracy: 60.68 -> 63.74 (+3.06) | Helped |
| Apply RestoreLCC instead of the strongest listed baseline | SlimGPT-pruned LLaMA-7B, task-specific recovery · Commonsense reasoning | mean accuracy: 56.15 -> 59.55 (+3.40) | Helped |
| Add contrastive probing for head selection rather than randomly selecting heads (ablation) | RestoreLCC on Wanda-pruned LLaMA-7B · Commonsense reasoning, general recovery | mean accuracy: 57.57 (-1.26) -> 58.83 | Helped |
| Select heads using contrastive probing rather than smallest output MSE (ablation) | RestoreLCC on Wanda-pruned LLaMA-7B · Commonsense reasoning, general recovery | mean accuracy: 58.14 (-0.69) -> 58.83 | Helped |
| Select heads using contrastive probing rather than smallest KL divergence (ablation) | RestoreLCC on Wanda-pruned LLaMA-7B · Commonsense reasoning, general recovery | mean accuracy: 57.92 (-0.91) -> 58.83 | Helped |
Surprising findings
- Discriminative information often resides in minor SVD components rather than the principal ones, per the authors' Finding 3.
- Compensating only FFN modules yields 57.00 mean accuracy, lower than compensating attention heads (58.83).
Limitations
- Learnable magnitudes for all components may overfit unimportant components.
- Only attention-head compensation is studied; FFN compensation performs worse in the authors' tests.
- No error bars or statistical significance reported due to limited compute.
Experimental setup
Datasets: Alpaca C4 BoolQ WikiText
Benchmarks: WikiText BoolQ RTE HellaSwag WinoGrande ARC-easy ARC-challenge OpenBookQA
Metrics: perplexity accuracy
Baselines: LoRA DoRA FLAP EoRA LoFiT Wanda SparseGPT SlimGPT full-parameter tuning
Builds on: LoRA DoRA LoFiT Wanda SparseGPT
Closest papers by meaning
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.90)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.89)
- PAT: Pruning-Aware Tuning for Large Language Models (AAAI 2025 · similarity 0.89)
- The Sparsity Whisperer (NeurIPS 2026 · similarity 0.89)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.89)
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.