Research map / Architectures and efficiency / Pruning and efficient tuning
TTE: Two Tokens Are Enough to Improve Parameter-Efficient Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies overfitting in parameter-efficient tuning (PET) of ViTs on VTAB-1K and few-shot FGVC, shows existing regularizers hurt PET, and proposes TTE, which adds two learnable tokens plus a parameter-free cross-attention loss.
Problem
Parameter-efficient tuning (PET) of pre-trained vision transformers overfits when downstream samples are scarce, and regularization methods designed for full fine-tuning do not transfer to PET.
Approach
Introduce a globally learnable token and an instance-specific token (with adaptive dropout) appended to the ViT sequence, and regularize with a parameter-free cross-attention loss that aligns the global token's attention output over the CLS token with its attention output over the instance token.
Key results
- On VTAB-1K, TTE improves VPT, S-Ada., P-Ada., and LoRA by an average of 3.36% while adding little parameter count.
- P-Ada.+TTE (0.15M params) reaches 88.4% mean on the five-dataset Table 3 subset versus DR-Tune's 87.4% with 85.84M params, a 1% gain.
- Existing regularizers hurt PET: DR-Tune drops VPT from 62.41% to 61.60%, and L1/L2/USKD also underperform vanilla on several PET methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TTE to P-Ada. | P-Ada. on VTAB-1K · VTAB-1K | Top-1 accuracy: 71.35 -> 75.24; 3.89↑ | Helped |
| Apply TTE to S-Ada. | S-Ada. on VTAB-1K · VTAB-1K | Top-1 accuracy: 71.55 -> 75.06; 3.51↑ | Helped |
| Apply TTE to VPT. | VPT on VTAB-1K · VTAB-1K | Top-1 accuracy: 62.41 -> 66.44; 4.03↑ | Helped |
| Apply TTE to LoRA. | LoRA on VTAB-1K · VTAB-1K | Top-1 accuracy: 70.43 -> 72.45; 2.02↑ | Helped |
| Apply TTE to P-Ada. | P-Ada. on FGVC few-shot datasets · FGVC | Top-1 accuracy: +2.27% (1-shot); +1.52% (2-shot); +1.35% (4-shot); +1.41% (8-shot); +1.22% (16-shot) | Helped |
| Add DR-Tune regularization to VPT. | VPT on VTAB-1K · VTAB-1K | Top-1 accuracy: 62.41 -> 61.60; 0.81% | Hurt |
| Add L1 penalty regularization. | PET methods on VTAB-1K · VTAB-1K | Mean accuracy: 62.41 -> 63.01; 71.55 -> 72.13; 71.35 -> 69.86; 70.43 -> 70.17 | Mixed |
| Add L2 penalty regularization. | PET methods on VTAB-1K · VTAB-1K | Mean accuracy: 62.41 -> 62.84; 71.55 -> 72.03; 71.35 -> 70.72; 70.43 -> 70.01 | Mixed |
| Add the global-token cross-entropy loss L2. (ablation) | TTE with P-Ada. on VTAB-1K · VTAB-1K | Top-1 accuracy: 71.38 -> 72.88 | Helped |
| Add PFCA loss L3. (ablation) | TTE with P-Ada. on VTAB-1K · VTAB-1K | Top-1 accuracy: 71.38 -> 74.30 | Helped |
| Use mean absolute error instead of mean squared error for PFCA loss. (ablation) | TTE with P-Ada. on VTAB-1K · VTAB-1K | Top-1 accuracy: 75.24 -> 74.19 | Hurt |
| Use cosine loss instead of mean squared error for PFCA loss. (ablation) | TTE with P-Ada. on VTAB-1K · VTAB-1K | Top-1 accuracy: 75.24 -> 73.27 | Hurt |
Surprising findings
- Standard regularizers (L1, L2, USKD, DR-Tune) generally degrade PET performance rather than helping.
- Increasing the token length of the learnable tokens decreases accuracy, suggesting that adding more parameters causes insufficient learning.
Limitations
- Regularization loss is specific to classification tasks (PFCA loss uses classification-head logits and CLS token).
- Hyperparameters are restricted (temperature fixed at 0.5, beta searched over three values), and results rely on ImageNet-21K ViT-B/16 backbone.
Experimental setup
Datasets: VTAB-1K FGVC-few-shot (Food-101, OxfordPets, Stanford Cars, Oxford-Flowers102, FGVC-Aircraft) ImageNet-21K
Benchmarks: VTAB-1K FGVC few-shot CIFAR-100 Caltech101 DTD Flowers102 Pets
Metrics: top-1 accuracy mean accuracy trainable parameters (M)
Baselines: VPT S-Ada. P-Ada. LoRA DR-Tune Core-tuning RLRR ARC Res-Tuning SPT LAST L1 penalty
Builds on: VPT AdaptFormer LoRA DR-Tune USKD
Closest papers by meaning
- Faster Parameter-Efficient Tuning with Token Redundancy Reduction (CVPR 2025 · similarity 0.88)
- Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods (ICML 2025 · similarity 0.87)
- Densely Connected Parameter-Efficient Tuning for Referring Image Segmentation (AAAI 2025 · similarity 0.87)
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.86)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.85)
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.