Research map / Vision-language and multimodal / CLIP and zero-shot
Domain-Controlled Prompt Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Domain-Controlled Prompt Learning (DCPL), which uses features from a large-scale specific-domain foundation model (LRSM for remote sensing, MedSAM for medical) to generate domain biases for CLIP's visual and language prompt branches, plus a noisy-adding strategy against overfitting.
Problem
Prompt learning for vision-language models like CLIP transfers poorly from natural images to specific domains such as remote sensing and medical images, and base-to-novel overfitting persists.
Approach
Lightweight control nets map specific-domain foundation model embeddings into domain biases added to both the learnable context vectors (language branch) and image features (visual branch), and Gaussian noise scaled by the mean image embedding is added to visual features during training to escape suboptimal solutions.
Key results
- On 8 remote sensing datasets, base-to-novel average HM 83.36 (ViT) vs MaPLe 80.42.
- UCM ablation: full DCPL reaches HM 88.09 vs MaPLe-style baseline 82.17.
- Medical average HM 53.08 vs MaPLe 49.01 across BTMRI, CHMNIST, CCBTM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add domain-controlled visual and language prompts using remote-sensing domain knowledge | CLIP prompt learning · RSI base-to-novel, average over 8 datasets | HM: 80.42 -> 83.36 | Helped |
| Use ViTAE instead of ViT for the remote-sensing foundation model (ablation) | DCPL · RSI base-to-novel, average over 8 datasets | HM: 83.36 -> 81.81 | Hurt |
| Add DCPL to prompt learning | CLIP prompt learning · RSI cross-dataset, average target accuracy | accuracy: 72.42 -> 73.46 | Helped |
| Add DCPL to prompt learning | CLIP prompt learning · RSI domain generalization, average target accuracy | accuracy: 72.15 -> 73.43 | Helped |
| Add DCPL to prompt learning | CLIP prompt learning · Medical base-to-novel, average over datasets | HM: 49.45 -> 53.08 | Helped |
| Add visual domain control (ablation) | DCPL on UCM · UCM | HM: 82.17 -> 85.80 | Helped |
| Add language domain control (ablation) | DCPL on UCM · UCM | HM: 82.17 -> 83.74 | Helped |
| Add visual and language domain control together (ablation) | DCPL on UCM · UCM | HM: 82.17 -> 88.09 | Helped |
| Replace noisy-adding with dropout at rate 0.3 (ablation) | DCPL on UCM · UCM | HM: 82.17 -> 86.67 | Helped |
| Replace noisy-adding with dropout at rate 0.5 (ablation) | DCPL on UCM · UCM | HM: 82.17 -> 86.38 | Helped |
| Replace noisy-adding with mutation rate 0.05 (ablation) | DCPL on UCM · UCM | HM: 82.17 -> 82.65 | Helped |
| Replace noisy-adding with mutation rate 0.1 (ablation) | DCPL on UCM · UCM | HM: 82.17 -> 82.44 | Helped |
Surprising findings
- Dropout (0.3) on UCM gives HM 86.67, which is competitive with the noisy-adding strategy's 88.09 gain over baseline.
- Deeper ViTAE backbone underperformed ViT backbone on average, suggesting an upper limit to using remote sensing foundation knowledge.
Limitations
- Gains on some datasets are small or negative (e.g., PatternNet, UCM in cross-dataset; CHMNIST novel, CCBTM base).
- Results rely on domain-specific foundation models (LRSM, MedSAM) that are not general across all specific domains.
- Ablations are run only on UCM, and results are averaged over three runs.
Experimental setup
Datasets: MLRSNet PatternNet RSSCN7 AID RSICD UCM WHURS19 NWPU BTMRI CHMNIST CCBTM
Benchmarks: MLRSNet PatternNet RSSCN7 AID RSICD UCM WHURS19 NWPU BTMRI CHMNIST CCBTM
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp MaPLe Dropout Mutation
Builds on: CoOp CoCoOp MaPLe CLIP MAE
Closest papers by meaning
- Domain Prompt Learning with Quaternion Networks (CVPR 2024 · similarity 0.90)
- Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation (CVPR 2024 · similarity 0.90)
- Enhancing Domain Adaptation through Prompt Gradient Alignment (NeurIPS 2024 · similarity 0.90)
- Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction (AAAI 2024 · similarity 0.89)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.89)
- Preserving Clusters in Prompt Learning for Unsupervised Domain Adaptation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.