Research map / Label-efficient and robust learning / Domain adaptation and generalization
Domain Generalization in CLIP via Learning with Diverse Text Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a CLIP-based domain generalization method that uses 80 CLIP prompt templates as imitated domains to suppress domain-sensitive channels, enforce prediction consistency, and diversify orthogonal feature components.
Problem
Domain generalization with CLIP is hampered by CLIP capturing domain-specific details in its image-text alignment and by limited feature diversity in the few source domains, which biases the adapted model.
Approach
Uses diverse CLIP template prompts as proxies for multiple domains to guide direct fine-tuning of the image encoder via three terms: entropy-filtered domain-sensitive channel suppression, a multi-domain KL consistency constraint, and orthogonal uniformity regularization, on top of a distillation baseline.
Key results
- Achieves 79.17% average accuracy across five DomainBed datasets with ViT-B/16, versus 77.73% for VL2V-SD (prior best), about 1.4 points better.
- Reaches 63.27% on TerraIncognita, a 4.7-point gain over the prior state of the art.
- Ablation on four datasets: full method reaches 83.19% average, versus 81.46% for the base distillation setup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the proposed diverse-text-prompt method | CLIP ViT-B/16 · TerraInc | accuracy: 59.30 -> 63.27 | Helped |
| add the proposed diverse-text-prompt method | CLIP ViT-B/16 · Office Home | accuracy: 87.38 -> 87.65 | Helped |
| add the proposed diverse-text-prompt method | CLIP ViT-B/16 · VLCS | accuracy: 84.30 -> 84.79 | Helped |
| add the proposed diverse-text-prompt method | CLIP ViT-B/16 · PACS | accuracy: 97.30 -> 97.03 | Hurt |
| add the proposed diverse-text-prompt method | CLIP ViT-B/16 · DomainNet | accuracy: 62.79 -> 63.11 | Helped |
| add domain-sensitive channel suppression with entropy-based prompt selection (ablation) | Base CLIP DG model · OH, TI, VLCS, PACS Avg | accuracy: 81.46 -> 82.28 | Helped |
| add multi-domain consistency constraint with entropy-based prompt selection (ablation) | Base CLIP DG model · OH, TI, VLCS, PACS Avg | accuracy: 81.46 -> 82.37 | Helped |
| add both channel suppression and consistency constraint (ablation) | Base CLIP DG model · OH, TI, VLCS, PACS Avg | accuracy: 81.46 -> 82.79 | Helped |
| add orthogonal uniformity regularization (ablation) | Base CLIP DG model · OH, TI, VLCS, PACS Avg | accuracy: 81.46 -> 82.30 | Helped |
| apply uniformity regularization directly to image features without orthogonal decomposition (ablation) | Base CLIP DG model · OH, TI, VLCS, PACS Avg | accuracy: 81.46 -> 81.55 | Hurt |
| combine EPS, DCS, MCC, and OUR (ablation) | Base CLIP DG model · OH, TI, VLCS, PACS Avg | accuracy: 81.46 -> 83.19 | Helped |
| use the proposed full method rather than single or ensembled prompt-only distillation (ablation) | CLIP ViT-B/16 DG model · OH, TI, VLCS, PACS Avg | accuracy: 81.9 -> 83.2 | Helped |
Surprising findings
- Ensembled CLIP template prompts for distillation perform slightly worse than the single manual prompt 'a {domain} of a {class}', so the gains come mainly from the three regularizers rather than prompt diversity alone.
Limitations
- Gains on some datasets are small (e.g., VLCS 84.79 vs. 84.30 for DPL and PACS 97.03 vs. 97.30 for DPL), and the method relies on hand-picked CLIP templates as domain proxies.
- Hyperparameters (m=40 prompts, 20% suppression ratio) are tuned on the reported benchmarks.
Experimental setup
Datasets: PACS VLCS TerraIncognita OfficeHome DomainNet
Benchmarks: PACS VLCS TerraIncognita OfficeHome DomainNet
Metrics: leave-one-domain-out accuracy average accuracy
Baselines: SWAD LP-FT WiSE-FT MIRO DART SAGM FLYP CLIPood RISE VL2V-SD DPL SPG
Builds on: CLIP VL2V-SD SWAD DomainBed
Closest papers by meaning
- CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment (NeurIPS 2024 · similarity 0.93)
- OSLoPrompt: Bridging Low-Supervision Challenges and Open-Set Domain Generalization in CLIP (CVPR 2025 · similarity 0.92)
- Unknown Prompt the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization (CVPR 2024 · similarity 0.92)
- When and How Does CLIP Enable Domain and Compositional Generalization? (ICML 2025 · similarity 0.91)
- Disentangled Prompt Representation for Domain Generalization (CVPR 2024 · similarity 0.90)
- Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.