Research map / Architectures and efficiency / Pruning and efficient tuning
Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A unifying empirical study of ~14 PEFT methods on ViT-B/16 (ImageNet-21K) and CLIP for visual recognition, with hyperparameter tuning, analyzing low-shot (VTAB-1K), many-shot, prediction diversity/ensembles, and robustness with WiSE.
Problem
Existing PEFT methods for vision are reported with inconsistent (often under-tuned) hyperparameters, and there is no systematic guidance on when to apply PEFT, which method to use, how PEFT behaves in many-shot regimes, and how it affects robustness to distribution shifts.
Approach
Fairly re-evaluate representative PEFT methods with systematic tuning of learning rate, weight decay, PEFT size and drop path rate, then analyze prediction diversity, many-shot scaling, and robustness, and extend weight-space ensembling (WiSE) to PEFT.
Key results
- On VTAB-1K (19 tasks), most PEFT methods perform similarly when tuned (relative std dev low, ~0.5-2.7%) and consistently beat linear probing and full FT; BitFit reaches 75.6% overall mean vs 76.9% for Pfeiffer/AdaptFormer-level methods.
- Different PEFT methods make diverse predictions (about 20% disagreement on DTD and Retinopathy, about 35% on DMLab), and majority-vote ensembling gives consistent gains over the worst PEFT method.
- In many-shot settings PEFT with 2-5% trainable parameters matches full FT, and on CIFAR-100 PEFT outperforms full FT; WiSE applied to PEFT improves both target and distribution-shift accuracy, and WiSE on full FT ends up highest.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Tune representative PEFT methods carefully, including method-specific parameters | PEFT methods · VTAB-1K | top-1 accuracy | Helped |
| Use PEFT instead of linear probing or full fine-tuning | ViT-B/16 ImageNet-21K · VTAB-1K, 19 tasks | top-1 accuracy: 76.0 vs 61.9; 76.0 vs 70.0 | Helped |
| Activate a nonzero drop path rate rather than setting it to 0 (ablation) | PEFT methods · VTAB-1K | top-1 accuracy | Helped |
| Increase PEFT tunable parameters to 2%-5% of the model (ablation) | PEFT methods · CIFAR-100, RESISC, and Clevr-Distance many-shot | accuracy: 2%-5% | Helped |
| Use PEFT rather than full fine-tuning | ViT-B/16 ImageNet-21K · CIFAR-100 many-shot | accuracy | Helped |
| Use PEFT rather than full fine-tuning | CLIP ViT-B/16 · ImageNet-(V2, S, R, A) | average distribution-shift accuracy: 42.5 -> 55.4; 42.5 -> 56.9 | Helped |
| Apply WiSE weight-space interpolation (ablation) | PEFT fine-tuned CLIP ViT-B/16 · ImageNet-1K and ImageNet distribution shifts | accuracy | Helped |
| Apply WiSE to full fine-tuning instead of PEFT (ablation) | CLIP ViT-B/16 · ImageNet-1K and ImageNet distribution shifts | accuracy | Helped |
| Ensemble diverse PEFT methods by majority vote | PEFT methods · VTAB-1K datasets | accuracy | Helped |
Surprising findings
- Simple methods such as BitFit and LayerNorm tuning, previously reported as inferior, perform competitively with state-of-the-art visual PEFT methods when properly tuned.
- Full fine-tuning with WiSE outperforms PEFT with WiSE on both target and distribution-shift accuracy, reversing PEFT's robustness advantage over full FT.
Limitations
- Study limited to ViT-based backbones and visual recognition tasks; findings may not transfer to other architectures or modalities.
- Explanations of why PEFT works are empirical hypotheses rather than a definitive theory.
- Ensembling and semi-supervised uses are only preliminary attempts.
Experimental setup
Datasets: VTAB-1K CIFAR-100 RESISC45 Clevr-Distance ImageNet-1K ImageNet-V2 ImageNet-R ImageNet-Sketch ImageNet-A
Benchmarks: VTAB-1K CIFAR-100 RESISC45 Clevr-Distance ImageNet-V2 ImageNet-R ImageNet-Sketch ImageNet-A
Metrics: top-1 accuracy relative standard deviation prediction overlap ranking frequency
Baselines: linear probing full fine-tuning VPT-Shallow VPT-Deep BitFit DiffFit LayerNorm tuning SSF Pfeiffer Adapter Houlsby Adapter AdaptFormer RepAdapter
Builds on: VPT LoRA Adapter WiSE-FT VTAB-1K
Closest papers by meaning
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.91)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.90)
- Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model (CVPR 2024 · similarity 0.89)
- WST: Wavelet-Based Multi-scale Tuning for Visual Transfer Learning (AAAI 2025 · similarity 0.89)
- Visual Fourier Prompt Tuning (NeurIPS 2024 · similarity 0.88)
- SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.