Research map / Label-efficient and robust learning / Domain adaptation and generalization
A Closer Look at the CLS Token for Cross-Domain Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Finds that leaving the ViT CLS token randomly initialized rather than loading source-trained weights improves cross-domain few-shot performance, explains this via CLS absorbing low-frequency domain information, and proposes CD-CLS to decouple domain information with domain tokens.
Problem
Cross-domain few-shot learning with ViT transfers poorly because source-trained parameters carry domain-specific information, and the role of the CLS token in this transfer was previously unexamined.
Approach
The CLS token, placed at the input layer and shared across all patches, absorbs low-frequency (domain) information. The method fixes the CLS token at random initialization during source training, adds pseudo-domain tokens (one per source class) to absorb domain information, and then finetunes the CLS token alone on the target domain.
Key results
- CD-CLS achieves 66.10 average 5-way 5-shot accuracy without finetuning and 67.54 with finetuning, versus FLoR's 65.48 and 66.83 (ViT-S).
- In 5-way 1-shot with transductive finetuning, CD-CLS* reaches 56.72 average, above MEM-FS+RDA* at 55.14.
- Not loading the CLS token improved target-domain 1-shot average from 52.22 to 52.81 for the baseline, with gains mostly on CropDisease and EuroSAT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| not load the source-trained CLS token for target-domain evaluation (ablation) | baseline CDFSL · CropDisease, EuroSAT, ISIC2018, ChestX; 5-way 1-shot | accuracy: +1.32, +0.94, +0.01, +0.06 | Mixed |
| add CLS decoupling during source-domain training (ablation) | ViT-S CDFSL · four target datasets, 5-way 5-shot average | accuracy: 63.89±0.14 -> 65.66±0.15 | Helped |
| add the orthogonal loss to CLS decoupling (ablation) | ViT-S CDFSL · four target datasets, 5-way 5-shot average | accuracy: 65.66±0.15 -> 66.10±0.17 | Helped |
| load the DINO-pretrained CLS token instead of random initialization (ablation) | ViT-S CDFSL · four target datasets, 5-way 5-shot average | accuracy: 66.10±0.17 -> 64.46±0.14 | Hurt |
| make the CLS token learnable during source-domain training (ablation) | ViT-S CDFSL · four target datasets, 5-way 5-shot average | accuracy: 66.10±0.17 -> 64.83±0.15 | Hurt |
| remove domain tokens (ablation) | ViT-S CDFSL · four target datasets, 5-way 5-shot average | accuracy: 66.10±0.17 -> 64.50±0.16 | Hurt |
| fix domain tokens at random initialization instead of learning them (ablation) | ViT-S CDFSL · four target datasets, 5-way 5-shot average | accuracy: 66.10±0.17 -> 65.37±0.15 | Hurt |
| randomly assign domain tokens to samples (ablation) | ViT-S CDFSL · four target datasets, 5-way 5-shot average | accuracy: 66.10±0.17 -> 64.38±0.15 | Hurt |
| decouple by prompt instead of the proposed domain-token method (ablation) | ViT-S CDFSL · four target datasets, 5-way 5-shot average | accuracy: 66.10±0.17 -> 64.50±0.14 | Hurt |
| finetune the CLS token on the target domain rather than fixing it (ablation) | ViT-S CDFSL · four target datasets, 5-way 1-shot average | accuracy: 54.24±0.11 -> 54.62±0.13 | Helped |
| apply CLS decoupling during target-domain finetuning (ablation) | ViT-S CDFSL · four target datasets, 5-way 1-shot average | accuracy: 54.62±0.13 -> 51.77±0.12 | Hurt |
| re-initialize the CLS token during source-domain training (ablation) | CD-CLS · four target datasets, 5-way 5-shot average | accuracy: 65.66 -> 65.31 | Hurt |
Surprising findings
- Not loading the source-trained CLS token harms source-domain accuracy yet consistently improves target-domain performance.
- Loading the DINO-pretrained CLS token or learning it during source training both reduce performance relative to the fixed random CLS token.
Limitations
- Evaluated on only four target datasets, which may not represent real-world scenarios (per the authors' broader impact section).
- Decoupling the CLS token is harmful for finetuning-based target adaptation where no domain gap exists, so the method requires separate treatment across settings.
Experimental setup
Datasets: miniImageNet CropDiseases EuroSAT ISIC2018 ChestX
Benchmarks: CropDiseases EuroSAT ISIC2018 ChestX
Metrics: 5-way 1-shot accuracy 5-way 5-shot accuracy CKA similarity
Baselines: MEM-FS StyleAdv MICM SDT FLoR PMF LDP-net GNN+AFA RDC MEM-FS+RDA
Builds on: DINO StyleAdv FLoR ProtoNet CKA similarity
Closest papers by meaning
- Random Registers for Cross-Domain Few-Shot Learning (ICML 2025 · similarity 0.92)
- Attention Temperature Matters in ViT-Based Cross-Domain Few-Shot Learning (NeurIPS 2024 · similarity 0.91)
- Revisiting Continuity of Image Tokens for Cross-domain Few-shot Learning (ICML 2025 · similarity 0.91)
- Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning (CVPR 2024 · similarity 0.89)
- Reconstruction Target Matters in Masked Image Modeling for Cross-Domain Few-Shot Learning (AAAI 2025 · similarity 0.88)
- The Devil is in Low-Level Features for Cross-Domain Few-Shot Segmentation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.