Research map / Vision-language and multimodal / CLIP and zero-shot
Vision-aware Multimodal Prompt Tuning for Uploadable Multi-source Few-shot Domain Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UMFDA, a decentralized edge-collaborative few-shot multi-source domain adaptation setting, and VAMP, a vision-aware multimodal prompt tuning method on frozen CLIP that outperforms prompt-tuning baselines on OfficeHome and DomainNet.
Problem
Multi-source few-shot domain adaptation (MFDA) is costly for edge devices, which must store and process trainable parameters, while labels in source domains are scarce and most data is unannotated.
Approach
Domain-specific prompts are learned per edge device on a frozen CLIP. Visual prompts perceive domain information and are projected into domain-specific text prompts (vision→text), and inter-device losses (text classifier consistency, text semantic diversity) enable collaboration without central training.
Key results
- VAMP reaches 85.4% average accuracy on OfficeHome at 3% labels and 85.5% at 6%, versus 82.3% for zero-shot CLIP and 84.1% for the best baseline MaPLe at 6%.
- On DomainNet, VAMP averages 85.5% (1-shot) and 86.1% (3-shot), compared with 84.2% zero-shot CLIP.
- The vision→text prompt projection outperforms text→vision (85.7 vs 85.2 average on OfficeHome at 3%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VAMP instead of MaPLe | ViT-B/16 CLIP on OfficeHome · OfficeHome 3% (average) | accuracy (%): 84.0±0.4 -> 85.4±0.2 | Helped |
| use VAMP instead of MaPLe | ViT-B/16 CLIP on OfficeHome · OfficeHome 6% (average) | accuracy (%): 84.1±0.3 -> 85.5±0.1 | Helped |
| use VAMP instead of DAPL | ViT-B/16 CLIP on DomainNet · DomainNet 1 shot (average) | accuracy (%): 85.0±0.4 -> 85.5±0.0 | Helped |
| use VAMP instead of DAPL | ViT-B/16 CLIP on DomainNet · DomainNet 3 shot (average) | accuracy (%): 85.3±0.2 -> 86.1±0.0 | Helped |
| project prompts vision→text instead of text→vision (ablation) | VAMP on OfficeHome · OfficeHome 3% (average) | accuracy (%): 85.2 -> 85.7 | Helped |
| project prompts vision→text instead of text→vision (ablation) | VAMP on OfficeHome · OfficeHome 6% (average) | accuracy (%): 85.2 -> 85.6 | Helped |
| add TSD loss (ablation) | VAMP on OfficeHome · OfficeHome 3% (average) | accuracy (%): 85.5 -> 85.7 | Helped |
| add TCC loss (ablation) | VAMP on OfficeHome · OfficeHome 3% (average) | accuracy (%): 85.3 -> 85.5 | Helped |
| add DDA loss (ablation) | VAMP on OfficeHome · OfficeHome 3% (average) | accuracy (%): 85.3 -> 85.5 | Helped |
| vary prompt length from 2 to 16 (ablation) | VAMP on OfficeHome · OfficeHome 3% (average) | accuracy (%): 85.3, 85.2, 85.4, 85.4 | No clear effect |
Surprising findings
- DAPL and MPA, which disentangle prompts, underperform in the few-shot UMFDA setting, with MPA falling below zero-shot CLIP on OfficeHome.
- Zero-shot CLIP is already strong, and several prompt-tuning baselines only modestly exceed it.
Limitations
- Evaluated only on OfficeHome and DomainNet with ViT-B/16 CLIP and a small number of edge models.
- Pseudo-labels come from zero-shot CLIP with a fixed confidence threshold (0.8), which may limit adaptation quality.
- Text-feature inter-class variance is reduced relative to DAPL, as the authors acknowledge.
Experimental setup
Datasets: OfficeHome DomainNet
Benchmarks: OfficeHome DomainNet
Metrics: accuracy standard deviation Mann-Whitney U test p-value
Baselines: Zero-shot CLIP CoOp VPT MaPLe DAPL MPA MSFAN PCS CDS SImpAl PMDA
Builds on: CLIP MaPLe CoOp DAPL MPA MFSAN
Closest papers by meaning
- Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation (CVPR 2024 · similarity 0.88)
- Source-Free Domain Adaptation with Frozen Multimodal Foundation Model (CVPR 2024 · similarity 0.87)
- Exploring the Better Multimodal Synergy Strategy for Vision-Language Models (AAAI 2025 · similarity 0.86)
- Prompt-Based Distribution Alignment for Unsupervised Domain Adaptation (AAAI 2024 · similarity 0.86)
- Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.86)
- Enhancing Domain Adaptation through Prompt Gradient Alignment (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.