Research map / Architectures and efficiency / Pruning and efficient tuning
Resource-Efficient Transformer Pruning for Finetuning of Large Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
RECAP prunes transformers with Taylor-based structured importance and updates only Fisher-selected weights in an iterative prune-finetune-update loop, cutting finetuning GPU memory by up to 65%.
Problem
Finetuning large transformers (ViTs, LLMs, Mask2Former) under limited GPU memory; existing pruning methods need full-model finetuning on GPU before pruning to retain accuracy.
Approach
Iteratively prune with first-order Taylor importance on CPU, then finetune only a Fisher-selected subset of the pruned model on GPU and write updates back to the CPU-held full model, so the full model never has to reside on GPU during finetuning.
Key results
- RECAP reduces GPU memory footprint by up to 65% versus post-finetune pruning and movement pruning while staying competitive in accuracy (ViT-large CIFAR100 at 33% pruning: 91.93% vs 92.41% for Post-FT).
- On Mask2Former, RECAP cuts memory by about 34% (Swin-base) and 40% (Swin-large) relative to full finetuning-based pruning methods, with near-zero mIoU gap on Cityscapes and KITTI.
- On GLUE with BERT-base at 50% pruning, RECAP uses 702 MB versus 1553 MB for Post-FT pruning while matching it on CoLA, MRPC and SST-2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use iterative RECAP pruning and masked finetuning instead of pre-finetune pruning | ViT-base · CIFAR100 | accuracy: 86.57 -> 88.34 | Helped |
| Use iterative RECAP pruning and masked finetuning instead of pre-finetune pruning | ViT-base · TinyImageNet | accuracy: 81.41 -> 83.83 | Helped |
| Use iterative RECAP pruning and masked finetuning instead of post-finetune pruning | Mask2Former with Swin-base · KITTI-dev | mIoU: 70.78 -> 70.82 | Helped |
| Use iterative RECAP pruning and masked finetuning instead of post-finetune pruning | BERT-base · SST-2 | accuracy: 92.20 -> 92.20 | No clear effect |
| Use iterative RECAP pruning and masked finetuning instead of post-finetune pruning | BERT-base · CoLA | Matthews Correlation Coefficient: 57.78 -> 57.67 | No clear effect |
| Use iterative RECAP pruning and masked finetuning instead of post-finetune pruning | BERT-base · MRPC | F1: 86.81 -> 86.55 | No clear effect |
| Use iterative RECAP pruning and masked finetuning instead of post-finetune pruning | BERT-base · QNLI | accuracy: 91.12 -> 90.74 | Hurt |
| Increase the pruning ratio to 66.6% (ablation) | RECAP with ViT-base · CIFAR100 | accuracy: 88.34 -> 72.90 | Hurt |
| Increase the pruning ratio to 66.6% (ablation) | RECAP with ViT-large · CIFAR100 | accuracy: 91.93 -> 85.65 | Hurt |
| Use a zero or too-low finetuning mask ratio (ablation) | RECAP · CIFAR100, TinyImageNet, and CoLA-dev | performance | Hurt |
| Use a too-high finetuning mask ratio (ablation) | RECAP · CIFAR100, TinyImageNet, and CoLA-dev | performance | Hurt |
Surprising findings
- Pre-finetune pruning shows an accuracy drop relative to RECAP, and updating all weights of the pruned model saturates the pruning process early, so masked updates are needed.
Limitations
- Structured pruning causes significant performance drops at pruning ratios of 66% or more.
- For Mask2Former, activation memory dominates and RECAP is less efficient than adapter methods, which give 0.4-1% higher mIoU with lower cost.
- Experiments are limited to ViT, Mask2Former and BERT-base; LLM finetuning is motivated but not evaluated in the main text.
Experimental setup
Datasets: CIFAR100 TinyImageNet Cityscapes KITTI GLUE
Benchmarks: CIFAR100 TinyImageNet Cityscapes KITTI GLUE
Metrics: accuracy mIoU MCC F1 GPU memory footprint (MB)
Baselines: Full-FT Head-FT LoRA-FT Pre-FT pruning Post-FT pruning Movement pruning
Builds on: Taylor expansion pruning (Molchanov et al.) Movement pruning LoRA Mask2Former
Closest papers by meaning
- ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models (ICLR 2024 · similarity 0.90)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.90)
- Collaborative Multi-Mode Pruning for Vision-Language Models (CVPR 2026 · similarity 0.89)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.89)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.89)
- The Need for Speed: Pruning Transformers with One Recipe (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.