Research map / Label-efficient and robust learning / Domain adaptation and generalization

Attention Temperature Matters in ViT-Based Cross-Domain Few-Shot Learning

NeurIPS 2024 · Label-efficient and robust learning › Domain adaptation and generalization · approach: Cross-domain few-shot with transformers · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that scaling ViT attention by a small temperature (down to 0, i.e., uniform attention) improves cross-domain few-shot transfer, and proposes Attention Abandonment during source training plus target-time attention temperature adjustment to boost ViT-S transferability.

Problem

Vision Transformers transfer poorly under large domain gaps in cross-domain few-shot learning (CDFSL); the query-key attention yields ineffective target-domain attention.

Approach

Attention temperature (even 0, giving uniform attention) improves target-domain accuracy but hurts source accuracy; the authors attribute this to query-key attention being discriminative but poorly transferable, and propose randomly abandoning query-key attention (setting its temperature to 0 with probability p=0.8) during source training, plus a fixed target-time temperature.

Key results

• AttnTemp (ViT-S, fine-tuned, transductive) reaches 57.23 average 5-way 5-shot accuracy, vs. 55.62 for FLoR* (ResNet10) and 55.14 for MEM-FS+RDA*.
• Without finetuning, AttnTemp achieves 54.12 average 5-way 1-shot accuracy, above FLoR (52.80) with ViT-S.
• Attention Abandonment alone gives a 2.47% average 5-way 5-shot improvement over the baseline in ablation (63.53 to 66.00 with adjustment, 66.62 full).

Measured effects reported in the paper

Surprising findings

• Setting attention to a uniform map (temperature 0) for all blocks improves target-domain accuracy, even though it removes query-key attention entirely.
• Learnable temperature and dropout-based variants give only marginal gains compared with the proposed global, stochastic temperature.

Limitations

• Evaluated on only four target domains (CropDiseases, EuroSAT, ISIC2018, ChestX), so generality across domains is untested.
• Abandoning all attention (p=1) hurts performance, so the method depends on tuning the abandonment probability.
• Interpretation rests on empirical visualizations and CKA analyses rather than theory.

Experimental setup

Datasets: miniImageNet CropDiseases EuroSAT ISIC2018 ChestX

Benchmarks: miniImageNet CropDiseases EuroSAT ISIC2018 ChestX

Metrics: 5-way 1-shot accuracy 5-way 5-shot accuracy domain similarity (CKA)

Baselines: StyleAdv FLoR SDT MEM-FS PMF GNN+AFA LDP-net RDC MICM Learnable Temp.

Builds on: DINO StyleAdv ProtoNet Dropout

Closest papers by meaning

• A Closer Look at the CLS Token for Cross-Domain Few-Shot Learning (NeurIPS 2024 · similarity 0.91)
• Random Registers for Cross-Domain Few-Shot Learning (ICML 2025 · similarity 0.91)
• Revisiting Continuity of Image Tokens for Cross-domain Few-shot Learning (ICML 2025 · similarity 0.90)
• Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning (CVPR 2024 · similarity 0.88)
• On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.87)
• Attention Transfer Is Not Universally Effective for Vision Transformers (NeurIPS 2026 · similarity 0.86)