Research map / Vision-language and multimodal / Visual question answering
When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs
Beidi Zhao, Wenlong Deng, Xinting Liao, Yushu Li, Nazim Shaikh, Yao Nie, Xiaoxiao Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MAD-RAG (dual question plus attention mixing) to vanilla RAG | Vanilla RAG · E-VQA, LLaVA-1.5-7B | accuracy (%): 53.89 -> 63.09 | Improved |
| add attention mixing on top of dual-question input (ablation) | MAD-RAG · OK-VQA, LLaVA-1.5-7B | accuracy (%): 69.65 -> 70.22 | Improved |
| add attention mixing on top of dual-question input (ablation) | MAD-RAG · OK-VQA, Qwen2.5-VL-7B | accuracy (%): 63.85 -> 64.91 | Improved |
| duplicate the question (dual-question) without attention intervention (ablation) | Vanilla RAG · OK-VQA, LLaVA-1.5-7B | accuracy (%): 65.46 -> 69.65 | Improved |
| duplicate the question (dual-question) without attention intervention (ablation) | Vanilla RAG · OK-VQA, Qwen2.5-VL-7B | accuracy (%): 63.80 -> 63.85 | No clear effect |
| place question after the retrieved context (Swap Q/C) (ablation) | Vanilla RAG · OK-VQA, LLaVA-1.5-7B | accuracy (%): 65.46 -> 68.17 | Improved |
| place question after the retrieved context (Swap Q/C) (ablation) | Vanilla RAG · OK-VQA, Qwen2.5-VL-7B | accuracy (%): 63.80 -> 61.92 | Hurt |
| add retrieved context (RAG) versus closed-book | LLaVA-1.5-7B · image attention ratio | image attention ratio: 15.77% -> 9.07% | Hurt |
| apply attention mixing weight alpha between 0.1 and 0.5 (ablation) | MAD-RAG · OK-VQA, E-VQA, InfoSeek, LLaVA-1.5-7B | accuracy | No clear effect |
| use MAD-RAG instead of ALFAR on Qwen2.5-VL-7B OK-VQA | ALFAR · OK-VQA, Qwen2.5-VL-7B | accuracy (%): 63.55 -> 64.91 | Improved |
| add VLM hallucination mitigation methods (VHR) to RAG | Vanilla RAG · E-VQA, Qwen2.5-VL-7B | accuracy (%): 82.21 -> 58.67 | Hurt |
Closest papers by meaning
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.90)
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.89)
- RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering (NeurIPS 2025 · similarity 0.89)
- SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge (NeurIPS 2024 · similarity 0.89)
- Retrieval-Augmented Perception: High-resolution Image Perception Meets Visual RAG (ICML 2025 · similarity 0.89)
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
