Research map / Vision-language and multimodal / Multimodal LLMs
Enhancing Retrieval-Augmented Large Vision Language Models via Knowledge Conflict Mitigation
Wenbin An, Jiahao Nie, Feng Tian, Mingxiang Cai, Yaqiang Wu, Xiaoqin Zhang, Shijian Lu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Addresses knowledge conflicts between retrieved multimodal context and LVLMs' parametric knowledge in multimodal RAG for knowledge-intensive tasks.
Problem
Retrieved contextual knowledge in multimodal retrieval-augmented generation is often misaligned with LVLMs' internal parametric knowledge, causing knowledge conflicts that degrade answers in knowledge-intensive tasks.
Approach
Mitigates knowledge conflicts between retrieved multimodal context and an LVLM's parametric knowledge to make retrieval-augmented LVLMs more reliable; the specific mechanism is not described in the available abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add KCM to retrieval-augmented decoding | LLaVA-1.5 · InfoSeek | accuracy: 51.97 -> 58.67 | Improved |
| add KCM | LLaVA-1.5 · OK-VQA | accuracy: 46.17 -> 60.90 | Improved |
| add KCM | LLaVA-1.5 · E-VQA | accuracy: 19.14 -> 29.95 | Improved |
| add KCM | LLaVA-1.5 · Human free-form | accuracy: 7.74 -> 12.42 | Improved |
| add KCM | InstructBLIP · Human free-form | accuracy: 4.02 -> 6.77 | Improved |
| add KCM | Shikra · InfoSeek | accuracy: 19.41 -> 27.90 | Improved |
| add KCM | MiniGPT-4 · InfoSeek | accuracy: 25.83 -> 31.29 | Improved |
| add KCM with ground-truth context | LLaVA-1.5 · InfoSeek | accuracy: 65.5 -> 71.2 | Improved |
| add KCM with ground-truth context | LLaVA-1.5 · InfoSeek | memorization ratio: 46.9 -> 39.7 | Improved |
| add KCM with noisier 2nd retrieved knowledge | LLaVA-1.5 · InfoSeek | accuracy: 38.8 -> 44.9 | Improved |
| compare KCM against best prior decoding method VCD | KCM · OK-VQA | accuracy: 57.80 -> 60.90 | Improved |
| add Attention Adjustment, Neuron Pruning, Context logits and Entropy weighting incrementally (ablation) | KCM · InfoSeek and AOK-VQA | VQA accuracy | Improved |
Closest papers by meaning
- MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.91)
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.91)
- Unified Knowledge Maintenance Pruning and Progressive Recovery with Weight Recalling for Large Vision-Language Models (AAAI 2025 · similarity 0.90)
- Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval (NeurIPS 2025 · similarity 0.90)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
