Research map / Large language models / Language, safety and interpretability
Sample-aware Adaptive Structured Pruning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaPruner, which jointly searches calibration data and Taylor-based importance metrics with Bayesian optimization (TPE) to improve structured pruning of LLMs, followed by LoRA fine-tuning.
Problem
Structured pruning of LLMs that relies on Taylor-expansion importance scores is sensitive to the choice of calibration data (randomly selected) and to a fixed single importance metric, degrading pruned model performance.
Approach
Build a joint solution space of calibration-data samples (from BookCorpus) and a mix of coarse-grained (weight vector) and fine-grained (weight element) Taylor importance metrics, then use Bayesian optimization (TPE) with pruned-model perplexity on WikiText2/PTB as the objective to adaptively select both.
Key results
- At 20% pruning ratio with LoRA fine-tuning on LLaMA-7B, AdaPruner reaches 61.44% average zero-shot accuracy, retaining about 97% of the unpruned model and beating LLM-Pruner-E2 by 1.37 points.
- Without fine-tuning at 20% on LLaMA-7B, AdaPruner reaches WikiText2 PPL 17.72 vs 19.09 for LLM-Pruner-E1.
- On Vicuna-7B at 20% with fine-tuning, AdaPruner averages 60.85%, 3.07 points above LLM-Pruner-E2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adaptively optimize calibration data and importance estimation metrics together | structured pruning of LLaMA-7B · 20% pruning, with fine-tuning; commonsense reasoning | average accuracy: +1.37% over LLM-Pruner-E2 | Helped |
| fix importance estimation to first-order Taylor metric instead of optimizing the metrics (ablation) | AdaPruner on LLaMA-7B · 20% pruning, with fine-tuning; commonsense reasoning | average accuracy: 60.61 -> 61.44 | Helped |
| fix importance estimation to second-order Taylor metric instead of optimizing the metrics (ablation) | AdaPruner on LLaMA-7B · 20% pruning, with fine-tuning; commonsense reasoning | average accuracy: 60.34 -> 61.44 | Helped |
| replace adaptive calibration-data search with random selection (ablation) | AdaPruner on LLaMA-7B · 20% pruning, with fine-tuning; commonsense reasoning | average accuracy: 60.73 -> 61.44 | Helped |
| replace joint optimization with combined optimization (ablation) | AdaPruner on LLaMA-7B · 20% pruning, with fine-tuning; commonsense reasoning | average accuracy: 60.98 -> 61.44 | Helped |
| use 10 independent calibration-data solution subspaces instead of the uniform solution space (ablation) | AdaPruner on LLaMA-7B · 20% pruning, with fine-tuning; commonsense reasoning | average accuracy: 60.51 -> 61.44 | Helped |
| select calibration samples by score instead of optimizing the full calibration-data space (ablation) | AdaPruner on LLaMA-7B · 20% pruning, with fine-tuning; commonsense reasoning | average accuracy: 60.38 -> 61.44 | Helped |
| replace Bayesian optimization with random search (ablation) | AdaPruner search process · pruning-solution search | iterations to convergence: about 300 -> less than 150 | Helped |
Surprising findings
- Randomly selected calibration data with the searched importance metric still outperforms LLM-Pruner-E1/E2 by 1.5 and 0.66 points, suggesting the searched metric is robust to calibration samples.
Limitations
- Search relies on pruned-model perplexity on WikiText2/PTB as the objective, which is a cheap proxy that may not reflect downstream accuracy.
- Evaluated only on LLaMA-7B and Vicuna-7B; no larger models or other architectures.
- Gains over strongest baselines are modest (e.g., 0.81 to 1.37 points average accuracy) and no variance across seeds beyond a p-value claim is reported.
Experimental setup
Datasets: BookCorpus Stanford Alpaca WikiText2 PTB
Benchmarks: WikiText2 PTB BoolQ PIQA HellaSwag WinoGrande ARC-e ARC-c OBQA
Metrics: perplexity zero-shot accuracy average accuracy parameter count latency
Baselines: Magnitude Wanda LLM-Pruner LoRAShear MoreauPruner Vector
Builds on: LLM-Pruner Tree-structured Parzen Estimator LoRA
Closest papers by meaning
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.92)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.92)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.91)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.91)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.91)
- Beware of Calibration Data for Pruning Large Language Models (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.