Research map / Label-efficient and robust learning / Domain adaptation and generalization
Source-Free Domain Adaptation with Frozen Multimodal Foundation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIFO, which adapts a source-free target model by alternately customizing a frozen CLIP via mutual-information prompt learning and distilling its knowledge into the target model with most-likely category encouragement and predictive consistency losses.
Problem
Source-free domain adaptation relies on pseudo-labels or auxiliary supervision from the source model and unlabeled target data, which are error-prone under domain shift; the paper asks whether off-the-shelf vision-language models can supply extra knowledge.
Approach
Keep CLIP frozen, learn only class prompts to customize it toward the target domain by maximizing mutual information with the target model's predictions, then distill this customized CLIP into the target model using a memory-aware most-likely-category loss plus a mutual-information predictive consistency loss, alternating across epochs.
Key results
- DIFO-C-B32 averages 92.5% on Office-31, 83.1% on Office-Home, 90.3% on VisDA-related Sy→Re, and 80.0% on DomainNet-126, beating the prior best methods in closed-set SFDA.
- Directly weighting source model and CLIP predictions never exceeds DIFO-C-B32 accuracy in the normalized weighting analysis.
- Partial-set and open-set SFDA on Office-Home reach 85.6% and 75.9% for DIFO-C-B32, improving over CoWA and CRS by 2.4% and 2.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DIFO-C-B32 instead of CoWA | Office-31 closed-set SFDA · Office-31 | average accuracy: 90.3 -> 92.5 | Helped |
| Use DIFO-C-B32 instead of TPDS | Office-Home closed-set SFDA · Office-Home | average accuracy: 73.5 -> 83.1 | Helped |
| Use DIFO-C-B32 instead of PLUE | VisDA closed-set SFDA · VisDA | accuracy: 88.3 -> 90.3 | Helped |
| Use DIFO-C-B32 instead of GKD | DomainNet-126 closed-set SFDA · DomainNet-126 | average accuracy: 68.7 -> 80.0 | Helped |
| Use DIFO-C-B32 instead of CoWA | Office-Home partial-set SFDA · Office-Home partial-set | average accuracy: 83.2 -> 85.6 | Helped |
| Use DIFO-C-B32 instead of CRS | Office-Home open-set SFDA · Office-Home open-set | average accuracy: 73.2 -> 75.9 | Helped |
| Add LTSC task-specific ViL customization (ablation) | DIFO-C-B32 · Office-31, Office-Home, VisDA | average accuracy: 62.3 -> 81.4 | Helped |
| Add LMCE most-likely category encouragement (ablation) | DIFO-C-B32 · Office-31, Office-Home, VisDA | average accuracy: 62.3 -> 82.4 | Helped |
| Add LPC predictive consistency (ablation) | DIFO-C-B32 with LTSC and LMCE · Office-31, Office-Home, VisDA | average accuracy: 85.1 -> 88.6 | Helped |
| Replace mutual-information alignment with KL divergence (ablation) | DIFO-C-B32 · Office-31, Office-Home, VisDA | average accuracy: 88.6 -> 87.0 | Hurt |
| Cancel prompt learning and use a fixed CLIP prompt (ablation) | DIFO-C-B32 · Office-31, Office-Home, VisDA | average accuracy: 88.6 -> 86.8 | Hurt |
| Add task-specific prompt-learning customization (ablation) | DIFO-C-B32 with fixed CLIP prompt · Office-31, Office-Home, VisDA | average accuracy: 86.8 -> 88.6 | Helped |
Surprising findings
- Naively averaging source-model and zero-shot CLIP predictions is far below DIFO, and no fusion weight beats DIFO, showing that a dedicated distillation scheme is needed.
Limitations
- Performance depends on the quality of the CLIP backbone, with the weaker ResNet variant performing below the ViT-B/32 variant.
- Relies on epoch-wise prediction banks and alternating updates whose sensitivity is not fully characterized.
Experimental setup
Datasets: Office-31 Office-Home VisDA DomainNet-126
Benchmarks: Office-31 Office-Home VisDA DomainNet-126
Metrics: classification accuracy
Baselines: Source CLIP Source+CLIP SHOT NRC GKD HCL AaD AdaCon CoWA SCLM ELR
Builds on: SHOT CLIP DAPL NRC CoWA
Closest papers by meaning
- Proxy Denoising for Source-Free Domain Adaptation (ICLR 2025 · similarity 0.90)
- Invertible Projection and Conditional Alignment for Multi-Source Blended-Target Domain Adaptation (AAAI 2025 · similarity 0.88)
- Understanding and Improving Source-free Domain Adaptation from a Theoretical Perspective (CVPR 2024 · similarity 0.88)
- Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.88)
- Unified Language-driven Zero-shot Domain Adaptation (CVPR 2024 · similarity 0.88)
- Revisiting Source-Free Domain Adaptation: Insights into Representativeness, Generalization, and Variety (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.