Research map / Vision-language and multimodal / Cross-modal learning
Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAGPT, a retrieval-augmented dynamic prompt-tuning framework that retrieves within-modality similar instances, reconstructs missing modality embeddings, and generates instance-specific prompts for a frozen pre-trained ViLT under incomplete multimodal inputs.
Problem
Robustness of pre-trained multimodal transformers to missing modalities, where dummy imputation adds noise and static, instance-agnostic prompts offer limited knowledge.
Approach
Retrieve similar instances via within-modality (multi-channel) similarity, recover missing modality embeddings by averaging retrieved same-modality embeddings refined by a learnable FFT-based filter, and generate context-aware dynamic text/vision/label prompts inserted into a frozen MMT.
Key results
- On MM-IMDb with 70% text missing, RAGPT reaches 55.16 F1-M and 55.00 F1-S, a 12.21% and 12.68% improvement over the best baseline.
- On HateMemes with 70% text missing, RAGPT reaches 64.10 AUROC versus 59.60 for the best baseline (MSPs).
- On Food101 with 70% text missing, RAGPT reaches 75.53 ACC versus 71.74 for MSPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RAGPT instead of the strongest baseline, MSPs | MM-IMDb · MM-IMDb, 70% text missing | F1-M: 49.16 -> 55.16 | Helped |
| Apply RAGPT instead of the strongest baseline, MSPs | MM-IMDb · MM-IMDb, 70% text missing | F1-S: 48.81 -> 55.00 | Helped |
| Apply RAGPT instead of the strongest baseline, MAPs | MM-IMDb · MM-IMDb, 70% image missing | F1-M: 44.86 -> 46.44 | Helped |
| Apply RAGPT instead of the strongest baseline, DrFuse | MM-IMDb · MM-IMDb, 70% both missing | F1-M: 48.83 -> 50.89 | Helped |
| Apply RAGPT instead of the strongest baseline, MSPs | HateMemes · HateMemes, 70% text missing | AUROC: 59.60 -> 64.10 | Helped |
| Apply RAGPT instead of the strongest baseline, DrFuse | HateMemes · HateMemes, 70% image missing | AUROC: 60.66 -> 62.57 | Helped |
| Apply RAGPT instead of the strongest baseline, ShaSpec | HateMemes · HateMemes, 70% both missing | AUROC: 60.96 -> 63.47 | Helped |
| Apply RAGPT instead of the strongest baseline, MSPs | Food101 · Food101, 70% image missing | ACC: 79.09 -> 81.98 | Helped |
| Use multi-channel retrieval rather than cross-modal retrieval (ablation) | RAGPT · MM-IMDb, 70% text missing | F1-M: 52.37 -> 55.16 | Helped |
| Add the retriever (ablation) | RAGPT · MM-IMDb, 70% text missing | F1-M: 49.25 -> 55.16 | Helped |
| Use the missing-modality generator instead of random-value padding (ablation) | RAGPT · MM-IMDb, 70% text missing | F1-M: 51.14 -> 55.16 | Helped |
| Add the generator filter (ablation) | RAGPT · MM-IMDb, 70% text missing | F1-M: 54.15 -> 55.16 | Helped |
Surprising findings
- Removing the filter layer in the missing modality generator gives a slight F1-M gain over the full model with text missing on MM-IMDb (54.15 vs 55.16 is lower, but F1-S drops only modestly), suggesting modest sensitivity to the filter in some metrics.
- Cross-modal retrieval (CLIP-based) underperforms within-modality retrieval, and the cross-modal retriever variant is worse than the full model.
Limitations
- Evaluated only on two-modality datasets (text and image) with a single frozen backbone (ViLT).
- Performance degrades when too many retrieved instances are used due to irrelevant noise, requiring tuning of K.
Experimental setup
Datasets: MM-IMDb HateMemes Food101
Benchmarks: MM-IMDb HateMemes Food101
Metrics: F1-Micro F1-Sample AUROC ACC
Baselines: SMIL TFR-Net AcMAE IF-MMIN ShaSpec DrFuse CorrKD MAPs MSPs
Builds on: ViLT MaPLe CLIP MAPs MSPs
Closest papers by meaning
- Deep Correlated Prompting for Visual Recognition with Missing Modalities (NeurIPS 2024 · similarity 0.89)
- Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement (AAAI 2025 · similarity 0.88)
- Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs (ICLR 2026 · similarity 0.88)
- Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning (NeurIPS 2026 · similarity 0.87)
- Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition (AAAI 2024 · similarity 0.87)
- Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.