Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Large-scale controlled study (~1.8k experiments) of parameter-efficient fine-tuning methods for few-shot ViT classification, proposing LN-TUNE and ATTNSCALE as strong baselines on META-DATASET and ORBIT.
Problem
Few-shot classification with fine-tuned ViTs is expensive in time, compute and storage, and prior PEFT results are hard to compare because experimental setups (feature extractor, pre-training, fine-tuning algorithm, hyperparameters) differ across papers.
Approach
Run a controlled comparison of seven existing PEFT methods plus full fine-tuning under identical settings, and show that tuning only LayerNorm parameters (LN-TUNE) is a strong baseline, while ATTNSCALE (learned per-element attention scaling plus eTT's domain-residual adapter) is strongest for self-supervised ViTs with about 9x fewer parameters than eTT.
Key results
- LN-TUNE fine-tunes under 0.08% of ViT-S/16 parameters and ranks top among PEFT methods on META-DATASET for supervised ViTs (e.g., 79.1 overall vs full fine-tuning 77.5 on ViT-S/16 DINO).
- ATTNSCALE reaches 79.4 average on META-DATASET (ViT-S/16 DINO, excluding ImageNet), the best among PEFT methods, while using about 1.2% of parameters, roughly 9x fewer than eTT.
- On ORBIT with ViT-S/16 (DINO), LN-TUNE gets 67.8 vs 65.6 for eTT and 63.1 for full fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add attention-matrix scaling with the DRA module instead of eTT prefix tuning | ViT-S/16 (DINO) few-shot adaptation · META-DATASET (excluding ImageNet) | accuracy: 78.1 -> 79.4 | Helped |
| fine-tune only LayerNorm parameters | ViT-S/16 (DINO) few-shot adaptation · META-DATASET (excluding ImageNet) | accuracy: 77.5 -> 79.1 | Helped |
| fine-tune only LayerNorm parameters | ViT-S/16 (DINO) few-shot adaptation · ORBIT | accuracy: 63.1 -> 67.8 | Helped |
| fine-tune only LayerNorm parameters | ViT-S/16 (DeiT) few-shot adaptation · ORBIT | accuracy: 66.6 -> 68.8 | Helped |
| use PROTOAUG instead of PROTONCC (ablation) | full fine-tuning with DINO · META-DATASET | accuracy: 72.2 -> 77.2 | Helped |
| use PROTOAUG instead of PROTONCC (ablation) | all PEFT methods with DINO · META-DATASET | accuracy: 73.2 -> 75.4 | Helped |
| use PROTOAUG instead of PROTONCC (ablation) | full fine-tuning with DeiT · META-DATASET | accuracy: 71.38 -> 78.1 | Helped |
| use PROTOAUG instead of PROTONCC (ablation) | all PEFT methods with DeiT · META-DATASET | accuracy: 68.4 -> 73.1 | Helped |
| share scaling parameters across attention heads (ATTNSCALELITE) rather than learn separate scaling parameters per head (ATTNSCALE) (ablation) | ViT-S/16 (DINO) few-shot adaptation · META-DATASET | accuracy: 79.4 -> 79.1 | Hurt |
Surprising findings
- Rankings of PEFT methods are inconsistent across pre-training objectives, and eTT, which was designed for self-supervised ViTs, does not generalize well to supervised ViTs.
- Deep prompt-tuning, the state-of-the-art PEFT on VTAB, falls short on META-DATASET, showing transfer-learning rankings do not carry over to few-shot settings.
Limitations
- Results are restricted to ViT-S/16 and ViT-B/16 pre-trained on ImageNet objectives and to two few-shot benchmarks.
- eTT's prototype regularization was omitted to keep comparisons fair, so eTT is not evaluated in its original form.
- ATTNSCALE's advantage is mainly for self-supervised ViTs; for supervised ViTs its margin over other methods shrinks on ORBIT and LN-TUNE is preferred.
Experimental setup
Datasets: META-DATASET ORBIT ImageNet-1k ImageNet-21k
Benchmarks: META-DATASET ORBIT
Metrics: few-shot classification accuracy
Baselines: full model fine-tuning adapters LoRA bias tuning ladder tuning shallow prompt-tuning deep prompt-tuning eTT
Builds on: eTT visual prompt tuning ProtoNets LoRA BitFit
Closest papers by meaning
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.91)
- Pushing the Limit of Fine-Tuning for Few-Shot Learning: Where Feature Reusing Meets Cross-Scale Attention (AAAI 2024 · similarity 0.90)
- ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.89)
- Be Confident in What You Know: Bayesian Parameter Efficient Fine-Tuning of Vision Foundation Models (NeurIPS 2024 · similarity 0.89)
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.88)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.