Research map / Vision-language and multimodal / Cross-modal learning
RAG4DMC: Retrieval-Augmented Generation for Data-Level Modality Completion
Ningxin He, Yongheng Deng, Sheng Yue, Yongjian Fu, Zhangzehui, Tiegang Gao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| complete missing modalities instead of dropping incomplete samples | CLIP/LLaVA training · MSCOCO (70% missing) | I2T R@1: 35.4 -> 41.4 | Improved |
| add retrieval grounding to generation (Vanilla-RAG vs Direct Generation) | generative completion · MSCOCO (70% missing) | I2T R@1: 41.4 -> 44.9 | Improved |
| naively merge external data into knowledge base | Vanilla-RAG · MSCOCO (70% missing) | I2T R@1: 44.9 -> 44.8 | No clear effect |
| add clustering filtering and cross-domain alignment of external data | Combined-RAG · MSCOCO (70% missing) | I2T R@1: 44.8 -> 45.9 | Improved |
| add multi-modal fusion retrieval | KFA-RAG · MSCOCO (70% missing) | I2T R@1: 45.9 -> 46.6 | Improved |
| use RAG baselines on remote-sensing data | Vanilla-RAG · RSICD | I2T R@1: 8.5 -> 4.8 | Hurt |
| add multi-modal fusion retrieval on RSICD | KFA-RAG · RSICD | I2T R@1: 5.3 -> 8.7 | Improved |
| use fusion score re-ranking instead of random top-k exemplar (ablation) | RAG4DMC · MSCOCO (70% missing) | CIDEr: 116.0 -> 117.0 | Improved |
| feed all retrieved exemplars to generator (ablation) | RAG4DMC · MSCOCO (70% missing) | I2T R@1: 46.5 -> 43.7 | Hurt |
| increase retrieval size from 10 to 15 (ablation) | RAG4DMC · MSCOCO (70% missing) | CIDEr: 117.2 -> 104.1 | Hurt |
| increase generation candidate number from 1 to 10 (ablation) | RAG4DMC · MSCOCO (70% missing) | CIDEr: 117.2 -> 121.5 | Improved |
| vary filtering thresholds 0.6-0.8 (ablation) | RAG4DMC · MSCOCO (70% missing) | CIDEr: 117.0 -> 117.2 | No clear effect |
Closest papers by meaning
- Knowledge Bridger: Towards Training-Free Missing Modality Completion (CVPR 2025 · similarity 0.89)
- Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning (AAAI 2025 · similarity 0.88)
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.87)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.86)
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.86)
- Gradient-Guided Modality Decoupling for Missing-Modality Robustness (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
