Research map / Architectures and efficiency / Pruning and efficient tuning
Adapters Strike Back
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies adapter position, inner structure, initialization and data normalization for ViTs, and proposes Adapter+ (post-position adapter with channel-wise learned scaling), achieving SOTA average accuracy on VTAB and FGVC.
Problem
Adapters for parameter-efficient transfer of vision transformers have often been reported to underperform methods like LoRA, partly due to unexamined design choices and implementation issues such as data normalization.
Approach
Systematic ablation of adapter position (Post-Adapter best), inner structure, and initialization, plus a learnable channel-wise scaling; combined with correct input data normalization and full training schedules, the resulting Adapter+ beats more complex PEFT methods on VTAB and FGVC with few parameters.
Key results
- Adapter+ reaches 77.6% average accuracy on VTAB test sets with 0.2M parameters without per-task hyperparameter tuning, 3.7 pp above previous adapter baselines.
- Adapter+ achieves 90.7% average on FGVC with 0.34M parameters under full schedule; 91.4% with test-set early stopping.
- Post-Adapter position yields best VTAB validation accuracy (76.0%) versus pre, intermediate, and parallel positions; channel-wise learned scaling adds +0.5 pp.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Adapter+ rather than the strongest reported competing methods | ViT-B/16 · VTAB test sets | average accuracy: 76.5 -> 77.6 | Helped |
| Choose rank per task from [1..32] instead of using a fixed rank (ablation) | Adapter+ · VTAB test sets | average accuracy: 77.6 -> 77.9 | Helped |
| Use Adapter+ instead of full fine-tuning | ViT-B/16 · FGVC test sets | average accuracy: 89.0 -> 90.7 | Helped |
| Place the adapter after the FFN (Post position) (ablation) | Adapterbase, rank 8 · VTAB validation sets | average accuracy: 75.3 -> 76.0 | Helped |
| Use the Pre position instead of the Post position (ablation) | Adapterbase, rank 8 · VTAB validation sets | average accuracy: 76.0 -> 75.3 | Hurt |
| Use the Intermediate position instead of the Post position (ablation) | Adapterbase, rank 8 · VTAB validation sets | average accuracy: 76.0 -> 75.1 | Hurt |
| Use the Parallel position instead of the Post position (ablation) | Adapterbase, rank 8 · VTAB validation sets | average accuracy: 76.0 -> 75.6 | Hurt |
| Keep biases in the adapter's linear layers (ablation) | Adapterbase · VTAB validation sets | average accuracy: 75.6 ± 0.4 -> 76.0 ± 0.2 | Helped |
| Use Houlsby rather than LoRA initialization (ablation) | Adapterbase · VTAB validation sets | average accuracy: 75.5 ± 0.3 -> 76.0 ± 0.2 | Helped |
| Use Houlsby rather than BERT initialization (ablation) | Adapterbase · VTAB validation sets | average accuracy: 75.8 ± 0.3 -> 76.0 ± 0.2 | Helped |
| Add channel-wise learned scaling (ablation) | Adapterbase · VTAB validation sets | average accuracy: 76.0 ± 0.2 -> 76.5 ± 0.2 | Helped |
| Add learned layer-wise scaling (ablation) | Adapterbase · VTAB validation sets | average accuracy: 76.0 ± 0.2 -> 76.2 ± 0.3 | Helped |
Surprising findings
- Adding layer normalization inside the adapter, as suggested by Pfeiffer et al., is detrimental in the ViT setting.
- Many prior VTAB results for adapters and VPT were depressed by unsuitable data normalization; correcting it improved VPT by 4.4 pp.
Limitations
- Several re-evaluated baselines (SSF, SPT, FacT) were run with released code, and Consolidator results were reported as-is due to missing code.
- Rank optimization per task uses the validation set and gives only a small gain (77.9%).
- Study restricted to ViT-B/16 pre-trained on ImageNet-21k and to image classification.
Experimental setup
Datasets: VTAB-1k CUB-200-2011 NABirds Oxford Flowers Stanford Dogs Stanford Cars ImageNet-21k
Benchmarks: VTAB-1k FGVC
Metrics: average accuracy number of trainable parameters
Baselines: LoRA VPT NOAH SSF FacT Consolidator SPT AdaptFormer Houlsby adapter Pfeiffer adapter full fine-tuning linear probing
Builds on: Houlsby adapters Pfeiffer adapters AdaptFormer LoRA VPT
Closest papers by meaning
- Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models (ICLR 2025 · similarity 0.86)
- WeGeFT: Weight‑Generative Fine‑Tuning for Multi‑Faceted Efficient Adaptation of Large Models (ICML 2025 · similarity 0.85)
- ProTransformer: Robustify Transformers via Plug-and-Play Paradigm (NeurIPS 2024 · similarity 0.85)
- Simplifying Transformer Blocks (ICLR 2024 · similarity 0.84)
- Adaptive Rank Allocation: Speeding Up Modern Transformers with RaNA Adapters (ICLR 2025 · similarity 0.84)
- MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.