Research map / Label-efficient and robust learning / Domain adaptation and generalization

Random Registers for Cross-Domain Few-Shot Learning

ICML 2025 · Label-efficient and robust learning › Domain adaptation and generalization · approach: Cross-domain few-shot with transformers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes REAP, which replaces clustered image tokens with random Gaussian register tokens during source-domain ViT training (sharpness-aware attention perturbation) and then fine-tunes learnable registers on the target domain, improving cross-domain few-shot learning.

Problem

Cross-domain few-shot learning with ViT: learnable prompt tuning on the source domain harms target-domain transfer, due to attention overfitting and sharper loss landscapes.

Approach

Random registers act as attention perturbations that approximate sharpness-aware minimization; REAP makes them more effective by replacing semantic clusters of image tokens with random noise, then switches to learnable registers for target-domain finetuning.

Key results

• REAP with ViT-S (finetuning, 5-shot) reaches 68.03 average accuracy across four target domains, above DAMIM's 67.55 and AttnTemp's 67.61 under the same setting.
• Without finetuning, REAP reaches 54.89 average in 1-shot, versus 54.12 for AttnTemp (NeurIPS-24).
• Ablation: source-domain REAP raises the 5-shot average from 64.07 (baseline) to 66.75, while random-mask and cluster-mask variants give 60.90 and 64.29.

Measured effects reported in the paper

Surprising findings

• Learnable prompts on the source domain reduce target-domain performance relative to random prompts, which consistently improve it, with the best results at the maximum number of random prompts GPU memory allows.
• Directly perturbing attention maps with Gaussian noise harms performance (attn_p 63.98 vs baseline 64.07 in 5-shot ablation).

Limitations

• Evaluated only on four target domains and a single source dataset (miniImageNet), which the authors acknowledge may not cover real-world scenarios.
• Explanation of sharpness and SAM connection is largely empirical and heuristic, with a loose theoretical link to SAM.

Experimental setup

Datasets: miniImageNet CropDiseases EuroSAT ISIC2018 ChestX

Benchmarks: CropDiseases EuroSAT ISIC2018 ChestX

Metrics: 5-way 1-shot accuracy 5-way 5-shot accuracy average accuracy

Baselines: MEM-FS StyleAdv FLoR DAMIM CD-CLS AttnTemp PMF MEM-FS + RDA LDP-net RDC

Builds on: Visual Prompt Tuning sharpness-aware minimization StyleAdv DINO prototype-based few-shot classification

Closest papers by meaning

• A Closer Look at the CLS Token for Cross-Domain Few-Shot Learning (NeurIPS 2024 · similarity 0.92)
• Attention Temperature Matters in ViT-Based Cross-Domain Few-Shot Learning (NeurIPS 2024 · similarity 0.91)
• Revisiting Continuity of Image Tokens for Cross-domain Few-shot Learning (ICML 2025 · similarity 0.91)
• Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning (CVPR 2024 · similarity 0.88)
• Task-Adaptive Prompted Transformer for Cross-Domain Few-Shot Learning (AAAI 2024 · similarity 0.88)
• Reconstruction Target Matters in Masked Image Modeling for Cross-Domain Few-Shot Learning (AAAI 2025 · similarity 0.87)