Research map / Vision-language and multimodal / CLIP and zero-shot
Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniMoS, which splits frozen CLIP visual features into language-associated and vision-associated components and trains them jointly via modality-ensemble training for unsupervised domain adaptation.
Problem
Unsupervised domain adaptation with vision-language models like CLIP, where prior methods adapt only the visual or only the textual branch and ignore the modality gap between the two.
Approach
Disentangle frozen CLIP-extracted visual features into language-associated and vision-associated components with orthogonal-regularized separators, distill CLIP zero-shot knowledge into the language part, train the vision part on a learnable-weight ensemble of both modalities, and align the two components across domains with a modality discriminator.
Key results
- UniMoS reaches 77.9% average accuracy on Office-Home with ResNet50, about 1.3 points above PADCLIP (76.6%) which fine-tunes the CLIP vision backbone.
- UniMoS reaches 88.1% on VisDA-2017 with ResNet101, behind PADCLIP (88.5%) but best among none-tuning CLIP methods.
- UniMoS trains about 47x faster than PADCLIP on VisDA-2017 while training only a few linear layers (0.79M params).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add orthogonal loss (ablation) | UniMoS · Office-Home (average) | accuracy: 77.4 -> 77.9 | Helped |
| Add orthogonal loss (ablation) | UniMoS · VisDA-2017 (average) | accuracy: 87.6 -> 88.1 | Helped |
| Add debiasing (ablation) | UniMoS · Office-Home (average) | accuracy: 77.3 -> 77.9 | Helped |
| Add debiasing (ablation) | UniMoS · VisDA-2017 (average) | accuracy: 87.7 -> 88.1 | Helped |
| Add information maximization loss (ablation) | UniMoS · Office-Home (average) | accuracy: 77.0 -> 77.9 | Helped |
| Add information maximization loss (ablation) | UniMoS · VisDA-2017 (average) | accuracy: 87.8 -> 88.1 | Helped |
| Add LAC distillation (ablation) | UniMoS · Office-Home (average) | accuracy: 77.0 -> 77.9 | Helped |
| Add LAC distillation (ablation) | UniMoS · VisDA-2017 (average) | accuracy: 87.6 -> 88.1 | Helped |
| Use a learnable ensemble weight instead of constant 0.5 (ablation) | UniMoS · Office-Home (average) | accuracy: 76.9 -> 77.9 | Helped |
| Use a learnable ensemble weight instead of constant 0.5 (ablation) | UniMoS · VisDA-2017 (average) | accuracy: 86.2 -> 88.1 | Helped |
| Add modality discriminator (ablation) | UniMoS · Office-Home (average) | accuracy: 77.6 -> 77.9 | Helped |
| Add modality discriminator (ablation) | UniMoS · VisDA-2017 (average) | accuracy: 87.9 -> 88.1 | Helped |
Surprising findings
- On DomainNet, UniMoS lags PADCLIP by 6.6% on average for the qdr target domain, attributed to a large domain gap that is hard to bridge without tuning CLIP backbones.
- Down-weighting source supervision on the language-associated branch helps on VisDA-2017, since CLIP struggles to identify synthetic source images.
Limitations
- Underperforms PADCLIP on VisDA-2017 and on the DomainNet qdr target, since CLIP backbones are not tuned and CLIP is not trained on synthetic images.
- Relies on frozen CLIP features, so it cannot adapt the encoder to large domain gaps.
Experimental setup
Datasets: Office-Home VisDA-2017 DomainNet Mini-DomainNet
Benchmarks: Office-Home VisDA-2017 DomainNet Mini-DomainNet
Metrics: classification accuracy
Baselines: CLIP DAPrompt ADCLIP PADCLIP ICON EIDCo FixBi CAN ATDOC MSGD
Builds on: CLIP PADCLIP DAPrompt ADCLIP deep clustering
Closest papers by meaning
- UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models (NeurIPS 2026 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation (CVPR 2024 · similarity 0.89)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.88)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.