Research map / Continual learning / Continual fine-tuning of LLMs
COPAL: Continual Pruning in Large Language Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COPAL, a training-free continual pruning method for LLMs that uses a sensitivity-based importance score accumulated across datasets to prune LLaMA models under sequential domain adaptation.
Problem
Pruning large language models while adapting them continually to new domains without retraining or forgetting previously encountered datasets; existing calibration-based post-training pruning suffers from weight stasis and forgetting when applied sequentially.
Approach
Accumulate a sensitivity-based importance score (magnitude of directional derivative of a finite-difference output perturbation loss, summed across datasets) so that weights relevant to all encountered datasets are retained, enabling pruning at each dataset step without retraining or storing past data.
Key results
- On LLaMA-7B at 50% unstructured sparsity, COPAL reduces average BWT to 0.016 versus 0.569 for Wanda and reaches average perplexity 9.728 versus 9.991 for Wanda.
- On LLaMA-65B at 50% unstructured sparsity, COPAL achieves average BWT of 0.001 and average PPL 6.791, compared with SparseGPT's 0.334 BWT and 7.235 PPL.
- Magnitude pruning under sequential initialization suffers weight stasis (BWT reported as WS), with average PPL of 30.246 on LLaMA-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COPAL instead of Wanda for 50% unstructured pruning | LLaMA-7B · Wikitext-2, PTB, C4 permutations (overall) | BWT: 0.569 ± 0.5556 -> 0.016 ± 0.017 | Helped |
| Use COPAL instead of Wanda for 50% unstructured pruning | LLaMA-7B · Wikitext-2, PTB, C4 permutations (overall) | PPL: 9.991 ± 2.541 -> 9.728 ± 2.221 | Helped |
| Use COPAL instead of Wanda for 50% unstructured pruning | LLaMA-13B · Wikitext-2, PTB, C4 permutations (overall) | BWT: 0.203 ± 0.065 -> 0.029 ± 0.028 | Helped |
| Use COPAL instead of SparseGPT for 50% unstructured pruning | LLaMA-13B · Wikitext-2, PTB, C4 permutations (overall) | PPL: 8.467 ± 1.766 -> 8.354 ± 1.851 | Helped |
| Use COPAL instead of Wanda for 50% unstructured pruning | LLaMA-30B · Wikitext-2, PTB, C4 permutations (overall) | BWT: 0.132 ± 0.043 -> 0.007 ± 0.008 | Helped |
| Use COPAL instead of Wanda for 50% unstructured pruning | LLaMA-30B · Wikitext-2, PTB, C4 permutations (overall) | PPL: 7.261 ± 1.595 -> 7.240 ± 1.576 | Helped |
| Use COPAL instead of Wanda for 50% unstructured pruning | LLaMA-65B · Wikitext-2, PTB, C4 permutations (overall) | BWT: 0.172 ± 0.221 -> 0.001 ± 0.078 | Helped |
| Use COPAL instead of SparseGPT for 50% unstructured pruning | LLaMA-65B · Wikitext-2, PTB, C4 permutations (overall) | PPL: 7.235 ± 1.971 -> 6.791 ± 1.666 | Helped |
| Increase the sparsity ratio (ablation) | COPAL · Continual pruning, previous tasks | BWT | No clear effect |
| Vary the number of calibration samples (ablation) | COPAL · Continual pruning | BWT | No clear effect |
Surprising findings
- Sequential application of calibration-based post-training pruning leads to weight stasis, where previously pruned weights remain zero and the model cannot adapt to new datasets.
- On LLaMA-65B with C4, COPAL's perplexity (7.036) is lower than the dense model's (5.978 average reported elsewhere is inconsistent; the table reports dense C4 PPL 5.978 vs COPAL 7.036), so the claim of enhanced comprehension is not clearly supported.
Limitations
- Evaluation limited to perplexity and BWT on three language-modeling datasets; no downstream task evaluation reported.
- Theoretical derivation relies on finite-difference approximations and heuristic absolute-value accumulation, with the gradient formulation not rigorously justified.
- Only uniform sparsity across linear layers is explored.
Experimental setup
Datasets: Wikitext-2 Penn Treebank (PTB) C4
Benchmarks: Wikitext-2 PTB C4
Metrics: perplexity backward transfer (BWT) average perplexity maximum perplexity
Baselines: Magnitude pruning SparseGPT Wanda
Builds on: Wanda SparseGPT Magnitude pruning
Closest papers by meaning
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.90)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.89)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.89)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.89)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.88)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.