Research map / Vision-language and multimodal / CLIP and zero-shot
Prompt Learning with Quaternion Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes domain prompt learning for CLIP that uses quaternion networks to fuse domain-specific foundation-model vision features with language prompts, and propagates the domain information to the vision branch, evaluated on remote sensing and medical image classification.
Problem
Adapting vision-language models (CLIP) to specialized domains such as remote sensing and medical imaging via prompt learning, where generalized VLMs misinterpret domain images and domain-specific foundation models are vision-only.
Approach
Places domain-specific vision features and language context embeddings on orthogonal axes of a quaternion hidden space to mine cross-modal relations, generating domain-specific text prompts; then uses learnable language prompts plus the domain features in quaternion layers to hierarchically generate vision prompts.
Key results
- Average base-to-novel harmonic mean over 8 remote sensing datasets reaches 83.50 (ViT backbone) vs 80.42 for MaPLe.
- On medical datasets, average base accuracy 74.36% vs 62.39% for MaPLe, with HM 53.36 vs 49.01.
- Cross-dataset transfer from MLRSNet averages 73.83% vs 72.42% for MaPLe; domain generalization averages 73.60%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use quaternion network to model cross-modal relationships (ablation) | Domain Prompt Learning · MLRSNet | HM: 75.87 -> 77.29 | Helped |
| prompt both vision and language branches rather than only language (ablation) | Domain Prompt Learning · MLRSNet | HM: 76.36 -> 77.29 | Helped |
| prompt both vision and language branches rather than only vision (ablation) | Domain Prompt Learning · MLRSNet | HM: 76.84 -> 77.29 | Helped |
| increase prompting depth from 3 to 6 (ablation) | Domain Prompt Learning · MLRSNet | HM: 73.99 -> 75.10 | Helped |
| increase prompting depth from 6 to 9 (ablation) | Domain Prompt Learning · MLRSNet | HM: 75.10 -> 77.29 | Helped |
| increase prompting depth from 9 to 12 (ablation) | Domain Prompt Learning · MLRSNet | HM: 77.29 -> 77.14 | Hurt |
| add noise to the language branch (ablation) | Domain Prompt Learning · MLRSNet | HM: 75.92 -> 77.29 | Helped |
| add noise to the vision branch (ablation) | Domain Prompt Learning · MLRSNet | HM: 75.92 -> 75.84 | Hurt |
| use domain-specific quaternion prompting rather than MaPLe | Domain Prompt Learning · Remote sensing base-to-novel, average over 8 datasets | HM: 80.42 -> 83.50 | Helped |
| use domain-specific quaternion prompting rather than MaPLe | Domain Prompt Learning · Medical base-to-novel, average over datasets | HM: 49.01 -> 53.36 | Helped |
| use domain-specific quaternion prompting rather than MaPLe | Domain Prompt Learning · Cross-dataset, MLRSNet source; average target accuracy | accuracy: 72.42 -> 73.83 | Helped |
| use domain-specific quaternion prompting rather than MaPLe | Domain Prompt Learning · Domain generalization, MLRSNet source; average target accuracy | accuracy: 72.15 -> 73.60 | Helped |
Surprising findings
- Adding noise to the vision branch decreased accuracy, which the authors attribute to disrupting well-matched vision-language relationships.
- The deeper ViTAE backbone did not outperform ViT overall, suggesting deeper architectures may hinder domain information extraction.
Limitations
- Limited gains on MLRSNet attributed to weak domain knowledge of the foundation model.
- Novel-class accuracy on medical datasets lags, suggesting domain foundation models do not yet generalize well to novel classes.
- Gains depend on availability of domain-specific foundation models, which are vision-only.
Experimental setup
Datasets: MLRSNet PatternNet RSSCN7 AID RSICD UCM WHURS19 NWPU BTMRI CCBTM CHMNIST
Benchmarks: MLRSNet PatternNet RSSCN7 AID RSICD UCM WHURS19 NWPU BTMRI CCBTM CHMNIST
Metrics: accuracy harmonic mean
Baselines: CLIP CoOp CoCoOp MaPLe ProGrad
Builds on: CLIP CoOp CoCoOp MaPLe quaternion neural networks
Closest papers by meaning
- Domain Prompt Learning with Quaternion Networks (CVPR 2024 · similarity 0.92)
- Deep Correlated Prompting for Visual Recognition with Missing Modalities (NeurIPS 2024 · similarity 0.86)
- QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization (NeurIPS 2024 · similarity 0.86)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.85)
- VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learning (NeurIPS 2025 · similarity 0.85)
- COMMA: Co-articulated Multi-Modal Learning (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.