Research map / Vision-language and multimodal / Cross-modal learning
Deep Correlated Prompting for Visual Recognition with Missing Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes deep correlated prompting (DCP), frozen CLIP with layer-correlated, input-dynamic, and modal-common prompts that adapt large multimodal models to missing-modality scenarios.
Problem
Large multimodal models degrade when input modalities are missing at training or test time, and full fine-tuning is computationally expensive.
Approach
Extends MMP's independent missing-aware prompts by generating each layer's prompts from preceding-layer prompts (correlated prompts), from the input features via self-attention (dynamic prompts), and splitting prompts into modal-common and modal-specific parts, while keeping the CLIP backbone frozen.
Key results
- On MM-IMDb at 70% missing rate, the method reaches 54.24 F1-Macro on validation versus 52.07-ish for MMP at 50% and clearly above MMP across all missing cases in the comparison table.
- Ours improves over MMP on all three datasets and missing cases at 50/70/90% missing rates in the comparison table (e.g., MM-IMDb 70% missing-both testing 53.14 vs MMP lower).
- Only about 4.0M extra parameters (2.4% of the 151M model) are tuned; with 0.2M parameters the method still reaches 53.14 F1 vs MMP 51.34 on MM-IMDb validation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply DCP instead of DePT | CLIP ViT-B/16 · MM-IMDb, η=70%, 65% image / 65% text, validation | F1-Macro: 53.17 -> 54.24 | Helped |
| apply DCP instead of DePT | CLIP ViT-B/16 · Food101, η=70%, 65% image / 65% text, validation | accuracy: 81.85 -> 82.38 | Helped |
| apply DCP instead of DePT | CLIP ViT-B/16 · Hateful Memes, η=70%, 65% image / 65% text, validation | AUROC: 63.32 -> 63.78 | Helped |
| add correlated prompts to independent layer prompts (ablation) | DCP on MM-IMDb · MM-IMDb validation | F1-Macro: 52.54 -> 54.24 | Helped |
| use an MLP instead of a fully connected layer for correlated prompts (ablation) | DCP on MM-IMDb · MM-IMDb validation | F1-Macro: 53.76 -> 54.24 | Helped |
| set correlated prompt depth to 6 rather than 3 or 12 (ablation) | DCP on MM-IMDb · MM-IMDb validation | F1-Macro: 53.74; 54.24; 53.86 | Mixed |
| use bimodal rather than unimodal information for correlated prompts (ablation) | DCP on MM-IMDb · MM-IMDb validation | F1-Macro: 53.52 -> 54.24 | Helped |
| set dynamic prompt depth to 1 rather than 2, 3, or 6 (ablation) | DCP on MM-IMDb · MM-IMDb validation | F1-Macro: 54.08; 53.76; 53.52 -> 54.24 | Helped |
| generate dynamic prompts with attention rather than max pooling and projection (ablation) | DCP on MM-IMDb · MM-IMDb validation | F1-Macro: 53.12 -> 54.24 | Helped |
| generate dynamic prompts with attention rather than min pooling and projection (ablation) | DCP on MM-IMDb · MM-IMDb validation | F1-Macro: 53.35 -> 54.24 | Helped |
| generate dynamic prompts with attention rather than average pooling and projection (ablation) | DCP on MM-IMDb · MM-IMDb validation | F1-Macro: 53.42 -> 54.24 | Helped |
| increase modal-common prompt depth from 1 to 6 (ablation) | DCP on MM-IMDb · MM-IMDb validation | F1-Macro: 54.24 -> 54.52 | Helped |
Surprising findings
- Missing text degrades performance more than missing image on MM-IMDb and Food-101, while missing image hurts more on Hateful Memes.
Limitations
- Only tested on two-stream multimodal models (CLIP; CoCa and ViLT only in appendix).
- Only two modalities evaluated.
- No error bars reported due to computational cost.
Experimental setup
Datasets: MM-IMDb UPMC Food-101 Hateful Memes
Benchmarks: MM-IMDb UPMC Food-101 Hateful Memes
Metrics: F1-Macro top-1 accuracy AUROC
Baselines: baseline (feature dropping, fc-only) CoOp MMP MaPLe DePT
Builds on: MMP CoOp MaPLe DePT CLIP
Closest papers by meaning
- Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning (AAAI 2025 · similarity 0.89)
- Exploring the Transferability of Visual Prompting for Multimodal Large Language Models (CVPR 2024 · similarity 0.89)
- Delving into Multimodal Prompting for Fine-Grained Visual Classification (AAAI 2024 · similarity 0.88)
- Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement (AAAI 2025 · similarity 0.88)
- ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts (CVPR 2024 · similarity 0.88)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.