Research map / Vision-language and multimodal / CLIP and zero-shot
Domain Prompt Learning with Quaternion Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes domain prompt learning for CLIP-style VLMs using quaternion networks that fuse domain-specific vision features from remote-sensing (ViT/ViTAE) and medical (MedSAM) foundation models into both language and vision prompts.
Problem
Adapting vision-language models via prompt learning to specialized domains (remote sensing, medical imaging) is underexplored, since natural-image prompt methods misinterpret domain images and domain-specific foundation models are vision-only and cannot prompt vision-language pairs.
Approach
Use quaternion networks to model orthogonal cross-modal relations between domain-specific vision features (from a domain foundation model) and language context embeddings, projecting generic language prompts into a specialized space, then hierarchically derive vision prompts from the language prompts and domain vision features; Gaussian noise is injected only in the language branch.
Key results
- Base-to-novel on 8 remote sensing datasets: average HM 83.50 with ViT backbone vs 80.42 for MaPLe (+3.08).
- Medical datasets average HM 53.36 vs 49.01 for MaPLe.
- Cross-dataset from MLRSNet to 8 targets: average 73.83 vs 72.42 for MaPLe; ablation shows removing quaternion network drops RSICD HM from 77.29 to 75.87.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use domain-specific prompting with the ViT backbone instead of MaPLe | Domain prompt learning · Remote sensing, average over 8 datasets | HM: 80.42 -> 83.50 | Helped |
| Use domain prompt learning instead of MaPLe | ViT-B/16 CLIP on medical datasets · Medical, average over datasets | HM: 49.01 -> 53.36 | Helped |
| Use domain prompt learning instead of MaPLe | ViT-B/16 CLIP · Cross-dataset generalization, average target | accuracy: 72.42 -> 73.83 | Helped |
| Use domain prompt learning instead of MaPLe | ViT-B/16 CLIP · Single-source multi-target domain generalization, average | accuracy: 72.15 -> 73.60 | Helped |
| Add the quaternion network (ablation) | Domain prompt learning on RSICD · RSICD | HM: 75.87 -> 77.29 | Helped |
| Prompt both vision and language branches instead of only the language branch (ablation) | Domain prompt learning on RSICD · RSICD | HM: 76.36 -> 77.29 | Helped |
| Prompt both vision and language branches instead of only the vision branch (ablation) | Domain prompt learning on RSICD · RSICD | HM: 76.84 -> 77.29 | Helped |
| Vary prompting depth from 3 through 12; depth 9 performs best (ablation) | Domain prompt learning on RSICD · RSICD | HM: 73.99; 75.10; 77.29; 77.14 | Mixed |
| Add noise in the language branch rather than omit noise or add it in the vision branch (ablation) | Domain prompt learning on RSICD · RSICD | HM: 75.92; 75.84 -> 77.29 | Helped |
Surprising findings
- Adding noise to the vision branch decreased accuracy, which the authors attribute to disrupting well-matched vision-language relations.
- Deeper ViTAE backbone did not outperform ViT overall, suggesting redundant parameters can hurt domain knowledge transfer.
Limitations
- Gains on some datasets are marginal (e.g., MLRSNet only +0.19 HM in base-to-novel) and the method does not top every dataset.
- On medical datasets, novel-class accuracy does not rank highest, suggesting domain models do not fully improve generalization.
- Evaluated only with 16-shot training on a single CLIP ViT-B/16 backbone setting in the main results.
Experimental setup
Datasets: MLRSNet PatternNet RSSCN7 AID RSICD UCM WHURS19 NWPU BTMRI CCBTM CHMNIST
Benchmarks: MLRSNet PatternNet RSSCN7 AID RSICD UCM WHURS19 NWPU BTMRI CCBTM CHMNIST
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp MaPLe ProGrad
Builds on: CoOp CoCoOp MaPLe quaternion neural networks
Closest papers by meaning
- Prompt Learning with Quaternion Networks (ICLR 2024 · similarity 0.92)
- Domain-Controlled Prompt Learning (AAAI 2024 · similarity 0.90)
- Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation (CVPR 2024 · similarity 0.89)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.89)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.88)
- Enhancing Domain Adaptation through Prompt Gradient Alignment (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.