Research map / Label-efficient and robust learning / Test-time adaptation
Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L2C, which adds a parallel revert-attention side network (CPNet) to a frozen CLIP and refines text features via greedy ensembling and uniformity loss, adapting both modalities with a domain prompt for few-shot test-time domain adaptation.
Problem
Few-shot test-time domain adaptation of CLIP using only a few unlabeled target images, where frozen-feature methods like VDPG are limited by CLIP's lack of dataset-specific knowledge and degrade on weaker backbones such as ViT-B/16.
Approach
Attach a lightweight parallel CPNet on the input space whose output is reverted-attention-masked so it learns only information absent from frozen CLIP; complement text features via greedy template ensembling plus a uniformity-refined text module; fuse adapted text and image features with a domain prompt built from batch-level attention and a K-V domain cache.
Key results
- On ViT-B/16, L2C reaches 35.2 Macro F1 on iWildCam (vs. VDPG 30.1, +5.1) and 40.9 WC Acc on FMoW (vs. VDPG 37.8, +3.1).
- On DomainNet with ViT-B/16, L2C averages 61.2% vs. VDPG 59.8%; with ViT-L/14 it averages 67.4% vs. VDPG 65.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CPNet to frozen zero-shot CLIP (ablation) | L2C · iWildCam | F1: 9.7 -> 22.8 | Helped |
| Add CPNet to frozen zero-shot CLIP (ablation) | L2C · FMoW | WC Acc: 14.5 -> 28.3 | Helped |
| Add text refinement (ablation) | L2C with CPNet · iWildCam | F1: 22.8 -> 24.0 | Helped |
| Add greedy text ensemble (ablation) | L2C with text refinement · iWildCam | F1: 24.0 -> 26.9 | Helped |
| Add revert attention (ablation) | L2C with text refinement and greedy ensemble · iWildCam | F1: 26.9 -> 32.9 | Helped |
| Add uniformity loss (ablation) | L2C with revert attention and text components · iWildCam | F1: 32.9 -> 33.2 | Helped |
| Add DAF (ablation) | L2C with ERM training · iWildCam | F1: 33.2 -> 32.8 | No clear effect |
| Use domain-centric training instead of ERM (ablation) | L2C with DAF · iWildCam | F1: 32.8 -> 35.2 | Helped |
| Include both K-V cache knowledge and the domain token in the domain prompt (ablation) | L2C · iWildCam | F1: 34.1 -> 35.2 | Helped |
| Use reshaped inter-attention for domain aggregation instead of mean aggregation (ablation) | L2C · FMoW | WC Acc: 37.8 -> 40.9 | Helped |
| Use greedy text ensemble instead of average text ensemble (ablation) | L2C · iWildCam | F1: 33.6 -> 35.2 | Helped |
| Use ATFD instead of uniformity loss for text inter-dispersion (ablation) | L2C · FMoW | WC Acc: 40.9 -> 40.8 | No clear effect |
Surprising findings
- Zero-shot CLIP regression on PovertyMap yields positive wealth-index correlation without any training.
Limitations
- CPNet must grow for downstream datasets that diverge strongly from CLIP pre-training (e.g., FMoW needs 6 blocks).
- Gains on ViT-L/14 are mixed; on some benchmarks (e.g., iWildCam Acc, FMoW Avg Acc) L2C is below VDPG or FLYP.
Experimental setup
Datasets: DomainNet iWildCam Camelyon17 FMoW PovertyMap
Benchmarks: DomainNet iWildCam Camelyon17 FMoW PovertyMap
Metrics: accuracy macro F1 worst-case accuracy Pearson r worst-case Pearson r
Baselines: VDPG Meta-DMoE MABN FLYP ERM MIRO DoPrompt CoOp CoCoOp DTL
Builds on: VDPG CLIP MetaDMoE MABN TPT
Closest papers by meaning
- OSLoPrompt: Bridging Low-Supervision Challenges and Open-Set Domain Generalization in CLIP (CVPR 2025 · similarity 0.89)
- Domain Generalization in CLIP via Learning with Diverse Text Prompts (CVPR 2025 · similarity 0.88)
- CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment (NeurIPS 2024 · similarity 0.88)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.88)
- Adapting to Distribution Shift by Visual Domain Prompt Generation (ICLR 2024 · similarity 0.88)
- Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.