Research map / Architectures and efficiency / Pruning and efficient tuning
Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes K-prune, a retraining-free structured pruning method for encoder PLMs (BERT, DistilBERT) that iteratively prunes sublayer-by-sublayer using knowledge measurement, knowledge-preserving mask search, and closed-form least-squares weight tuning.
Problem
Retraining-free structured pruning of pretrained encoder-based language models suffers severe accuracy degradation at high compression rates because pruning errors accumulate across layers.
Approach
Treat pruning error as loss of predictive and representational knowledge, measure each attention head's and neuron's knowledge, prune iteratively from bottom to top sublayer with a global-importance mask search, and reconstruct lost knowledge after each sublayer by solving a linear least-squares problem for the output projections.
Key results
- Up to 58.02%p higher F1 than existing retraining-free pruning methods at 80% FLOPs compression on SQuAD.
- Up to 2.93x inference speedup over the uncompressed model with under 3%p accuracy drop, versus at most 2.10x for competitors.
- Up to 422x lower pruning cost than retraining-based methods while matching or exceeding their accuracy on MNLI and QNLI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use K-prune instead of existing retraining-free pruning algorithms at 80% compression | Encoder-based pretrained models · SQuAD | F1: up to 58.02%p higher F1 score | Helped |
| Use K-prune rather than the uncompressed model | Encoder-based pretrained models · MRPC, STS-B, SQuAD 1.1, SQuAD 2.0 | inference speed: 2.66×; 2.43×; 2.60×; 2.93× | Helped |
| Add knowledge-based predictive and representational importance criteria (ablation) | K-prune · MRPC (80% compression) | accuracy: 84.07 -> 84.80 | Helped |
| Add knowledge-based predictive and representational importance criteria (ablation) | K-prune · SQuAD 1.1 (80% compression) | F1: 72.55 -> 74.16 | Helped |
| Add KPMS global-importance mask search instead of uniform pruning across sublayers (ablation) | K-prune · MRPC (80% compression) | accuracy: 81.71 -> 84.80 | Helped |
| Add KPMS global-importance mask search instead of uniform pruning across sublayers (ablation) | K-prune · SQuAD 1.1 (80% compression) | F1: 67.10 -> 74.16 | Helped |
| Add iterative pruning and weight-tuning (KPWT) (ablation) | K-prune · MRPC (80% compression) | accuracy: 68.38 -> 84.80 | Helped |
| Add iterative pruning and weight-tuning (KPWT) (ablation) | K-prune · SQuAD 1.1 (80% compression) | F1: 16.50 -> 74.16 | Helped |
| Prune OPT-1.3B at 5% or 10% | OPT-1.3B with K-prune · Wiki-text2 | perplexity: 14.67 -> 14.41; 14.67 -> 13.96 | Helped |
| Prune OPT-1.3B at 20% | OPT-1.3B with K-prune · Wiki-text2 | perplexity: 14.67 -> 15.74 | Hurt |
| Prune OPT-2.7B at 5%, 10%, or 15% | OPT-2.7B with K-prune · Wiki-text2 | perplexity: 12.46 -> 12.23; 12.46 -> 11.94; 12.46 -> 12.01 | Helped |
| Prune OPT-2.7B at 20% | OPT-2.7B with K-prune · Wiki-text2 | perplexity: 12.46 -> 12.51 | Hurt |
Surprising findings
- Removing the knowledge-preserving weight-tuning causes the largest accuracy drop (SQuAD F1 falls from 74.16 to 16.50 at 80% compression).
- On OPT-2.7B, perplexity rises only 0.40% at 20% pruning rate, and at 5-10% pruning perplexity actually decreases.
Limitations
- Evaluated only on encoder-based models (BERT, DistilBERT) in the main experiments; decoder LLM results are limited to OPT-1.3B/2.7B perplexity.
- Relies on a small sample dataset (100K tokens) and hyperparameters (gamma, lambda, mu) that need sensitivity tuning.
- Reimplemented KCM because no public implementation was available, which may affect the fairness of comparison.
Experimental setup
Datasets: MRPC MNLI QNLI SQuAD v1.1 SQuAD v2.0 GLUE C4
Benchmarks: GLUE SQuAD v1.1 SQuAD v2.0 Wiki-text2
Metrics: accuracy F1 score Spearman correlation inference speedup pruning time perplexity
Baselines: Kwon et al. retraining-free pruning KCM DynaBERT EBERT
Builds on: Kwon et al. retraining-free pruning CoFi DynaBERT EBERT
Closest papers by meaning
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.91)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.90)
- You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning (ICLR 2025 · similarity 0.90)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.90)
- The LLM Surgeon (ICLR 2024 · similarity 0.89)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.