Research map / Label-efficient and robust learning / Domain adaptation and generalization
Adapting to Distribution Shift by Visual Domain Prompt Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VDPG, which adapts a frozen CLIP image encoder to unseen target domains by generating a domain-specific visual prompt from few-shot unlabeled data via a knowledge bank, cross-attention generator, and meta-learned episodic training.
Problem
Few-shot test-time domain adaptation: adapting a model to an unseen target domain using only a few unlabeled target images, without degrading the out-of-distribution generalization of foundation models.
Approach
A shared learnable knowledge bank is condensed by a cross-attention domain prompt generator, conditioned on a small batch of target images, into a domain-specific visual prompt that guides frozen CLIP features via a guidance module; trained with a domain-aware soft-nearest-neighbor contrastive loss and episodic meta-learning.
Key results
- On WILDS iWildCam, VDPG reaches 78.8 accuracy and 46.5 macro F1 under OOD, outperforming FLYP (72.2 / 41.9) and Meta-DMoE/MABN baselines.
- On DomainNet with CLIP ViT-B/16, VDPG averages 59.8 accuracy versus 54.0 for MIRO and 44.4 for ERM.
- On Camelyon17, VDPG achieves 96.0 accuracy versus 92.4 for MABN, the best previous result.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VDPG rather than FLYP | CLIP ViT-L/14 · iWildCam | Macro F1: 41.9 (0.3) -> 46.5 (0.3) | Helped |
| Use VDPG rather than MABN | Camelyon17 · Camelyon17 | accuracy: 92.4 (1.9) -> 96.0 (0.4) | Helped |
| Use VDPG rather than FLYP | CLIP ViT-L/14 · FMoW | Avg Acc: 63.3 (0.4) -> 61.9 (0.4) | Hurt |
| Use VDPG rather than the strongest CNN baseline | PovertyMap · PovertyMap | Pearson r: 0.84 (0.05) -> 0.83 (0.04) | Hurt |
| Use VDPG rather than zero-shot CLIP | CLIP ViT-B/16 · DomainNet | Avg. accuracy: 57.1 -> 59.8 | Helped |
| Add Generator G to Frozen CLIP + GM under ERM training (ablation) | VDPG · iWildCam OOD | F1: 39.6 -> 32.5 | Hurt |
| Add the knowledge bank to Frozen CLIP + GM + Generator G under ERM training (ablation) | VDPG · iWildCam OOD | F1: 32.5 -> 32.4 | No clear effect |
| Use episodic rather than ERM training for the full model (ablation) | VDPG · iWildCam OOD | F1: 32.4 -> 43.6 | Helped |
| Add L_corr (ablation) | VDPG · iWildCam | F1: 54.5 -> 56.7 (ID); 43.6 -> 44.1 (OOD) | Helped |
| Add L_dac (ablation) | VDPG · iWildCam | F1: 54.5 -> 58.6 (ID); 43.6 -> 44.6 (OOD) | Helped |
| Add both L_corr and L_dac (ablation) | VDPG · iWildCam | F1: 54.5 -> 58.9 (ID); 43.6 -> 45.5 (OOD) | Helped |
| Train the generator and knowledge bank on unlabeled data (ablation) | VDPG with both losses · iWildCam | F1: 58.9 -> 60.2 (ID); 45.5 -> 46.5 (OOD) | Helped |
Surprising findings
- Adding the domain prompt generator and knowledge bank under standard ERM training decreases performance (iWildCam OOD F1 falls from 39.6 to 32.4), and only episodic training makes them helpful.
- Zero-shot CLIP collapses on iWildCam (1.0 macro F1), showing that source-domain learning is required.
Limitations
- Performance depends heavily on the out-of-distribution generalization of the foundation model.
- Foundation models may not cover data that is privacy-protected or otherwise unavailable.
- Hyperparameters such as sampling sizes were searched on iWildCam OOD validation and reused for other datasets.
Experimental setup
Datasets: WILDS-iWildCam WILDS-Camelyon17 WILDS-FMoW WILDS-PovertyMap DomainNet
Benchmarks: WILDS-iWildCam WILDS-Camelyon17 WILDS-FMoW WILDS-PovertyMap DomainNet
Metrics: accuracy Macro F1 worst-case accuracy Pearson correlation worst-case Pearson correlation
Baselines: ERM CORAL IRM ARM Meta-DMoE MABN Zero-shot CLIP FLYP DoPrompt MIRO Mixup MTL
Builds on: CLIP Visual Prompt Tuning Meta-DMoE FLYP soft-nearest neighbor loss
Closest papers by meaning
- Prompt-Based Distribution Alignment for Unsupervised Domain Adaptation (AAAI 2024 · similarity 0.89)
- A General Representation-Based Approach to Multi-Source Domain Adaptation (ICML 2025 · similarity 0.89)
- Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation (CVPR 2024 · similarity 0.89)
- Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization (AAAI 2024 · similarity 0.88)
- Preserving Clusters in Prompt Learning for Unsupervised Domain Adaptation (CVPR 2025 · similarity 0.88)
- Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.