Research map / Label-efficient and robust learning / Domain adaptation and generalization
Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multimodal unsupervised domain generalization (MUDG), retrieving task-relevant images from a large unlabeled web dataset (LAION-2B) via paired k-means indexing, adaptive text augmentation and a diversity-preserving loss to finetune CLIP.
Problem
Domain generalization when only task-agnostic unlabeled source data is available and target label names are the only task supervision; cross-modal approximate nearest neighbor search from text queries to images suffers low recall due to the modality gap.
Approach
Paired k-means clusters with centroids maintained in text-query space so that text queries fall in the same Voronoi cell as their nearest image; combined with an unsupervised, LLM-free adaptive label augmentation that keeps text feature variance, and retrieval-then-finetune with a diversity-preserving KL-regularized loss.
Key results
- On 11 zero-shot datasets with ViT-B/16, MUDG reaches 71.1% mean accuracy vs 69.0% for SuS-X and 65.2% for CLIP zero-shot.
- On ImageNet domain generalization benchmarks with ViT-B/16, MUDG achieves 61.4% mean vs 59.4% for the best ZS ensemble; on DomainNet 61.2% vs 60.2%.
- Paired k-means improves cross-modal R@1 over standard IVF k-means at the same probe count.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Paired k-means instead of standard image-centroid k-means for IVF index (ablation) | cross-modal nearest neighbor index · 11 datasets / ImageNet | mean accuracy: 71.1 vs 70.9 (11 datasets); 61.4 vs 61.2-equivalent ablation rows, small gain | Helped |
| Adaptive text augmentation instead of waffleCLIP augmentation (ablation) | MUDG training set construction · 11 datasets | mean accuracy: 71.1 vs 71.1 (paired k-means + loss with waffleCLIP) | No clear effect |
| Diversity preserving loss instead of vanilla cross entropy (ablation) | MUDG finetuning · 11 datasets | mean accuracy: 71.1 vs 70.9 | Helped |
| Diversity preserving loss vs contrastive loss and soft pseudo labels (ablation) | MUDG finetuning · 11 datasets | mean accuracy: 71.1 vs 70.9 (contrastive) and 69.5 (soft pseudo labels) | Helped |
| Clustering-based sample selection instead of no selection or random selection (ablation) | MUDG retrieved dataset · 11 datasets | mean accuracy: 71.1 vs 70.7 (none) and 70.9 (random) | Helped |
| Full MUDG pipeline vs CLIP zero-shot baseline | CLIP ViT-B/16 · 11 datasets | mean accuracy: 71.1 vs 65.2 | Helped |
| Full MUDG pipeline vs SuS-X on 11 datasets | CLIP ViT-B/16 · 11 datasets | mean accuracy: 71.1 vs 69.0 | Helped |
| Full MUDG pipeline vs PromptStyler (SFDG) on DomainNet | CLIP ViT-B/16 · DomainNet | mean accuracy: 61.2 vs 59.4 | Helped |
| Paired k-means, adaptive augmentation, and diversity loss together vs baseline k-means index with waffleCLIP and CE (ablation) | CLIP ViT-B/16 · ImageNet domains | mean accuracy: 61.4 vs 61.1 | Helped |
Surprising findings
- Adaptive text augmentation based on preserving label feature variance outperforms LLM-generated visual descriptors as query augmentations.
- Nearest-neighbor retrieval alone gives poor results on some datasets (e.g., EuroSAT 41.2% for ViT-B/16), below zero-shot.
Limitations
- Fails when target visual concepts are absent from the source dataset or misaligned with language in CLIP space (e.g., SAR, medical images).
- No gains on saturated zero-shot datasets such as VLCS.
Experimental setup
Datasets: LAION-2B-en
Benchmarks: ImageNet Caltech-101 Oxford-Pets Stanford-Cars Flowers-102 Food-101 FGVC-Aircraft SUN-397 DTD EuroSAT UCF-101 ImageNet-V2
Metrics: top-1 accuracy recall@1
Baselines: CLIP zero-shot waffleCLIP Random descriptors Visual descriptors CuPL SuS-X PromptStyler Herding K-center greedy
Builds on: SuS-X waffleCLIP FAISS PromptStyler React
Closest papers by meaning
- Reasoning-Driven Multimodal LLM for Domain Generalization (ICLR 2026 · similarity 0.90)
- Domain Generalization in CLIP via Learning with Diverse Text Prompts (CVPR 2025 · similarity 0.89)
- PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization (CVPR 2024 · similarity 0.89)
- Towards Generalizing to Unseen Domains with Few Labels (CVPR 2024 · similarity 0.89)
- Minimal Semantic Sufficiency Meets Unsupervised Domain Generalization (NeurIPS 2025 · similarity 0.89)
- Rethinking Multi-domain Generalization with A General Learning Objective (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.