Research map / Vision-language and multimodal / Visual question answering
MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MRAG-Bench, a vision-centric benchmark of 1,353 human-annotated multiple-choice questions over 9 scenarios with 16,130 images, showing LVLMs benefit more from retrieved images than text.
Problem
Existing multimodal retrieval-augmented benchmarks are text-centric; there is no systematic benchmark for evaluating whether LVLMs can leverage retrieved visual knowledge, which is more useful in certain scenarios (viewpoint, partial, transformative changes).
Approach
Categorizes scenarios where visual retrieval beats textual retrieval into perspective and transformative aspects (9 scenarios total) and evaluates LVLMs with query images plus retrieved or ground-truth image examples, comparing against text RAG.
Key results
- GPT-4o reaches 68.68% accuracy without RAG and 74.50% with ground-truth image RAG, while humans gain 33.16% with GT RAG versus GPT-4o's 5.82%.
- All evaluated LVLMs gain more from ground-truth image knowledge than from ground-truth text knowledge; e.g., LLaVA-NeXT-Interleave gains 11.90% more from images than text when GT RAG is used.
- Open-source models mostly decline with CLIP-retrieved images (e.g., Mantis-8B-siglip drops 45.01 to 39.62) while proprietary models such as GPT-4o still improve slightly (68.68 to 68.96).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ground-truth image RAG | GPT-4o · MRAG-Bench | accuracy: +5.82 | Helped |
| add retrieved image RAG | GPT-4o · MRAG-Bench | accuracy: +0.28 | No clear effect |
| add retrieved image RAG | LLaVA-OneVision · MRAG-Bench | accuracy: -3.18 | Hurt |
| add ground-truth image RAG | LLaVA-OneVision · MRAG-Bench | accuracy: +5.69 | Helped |
| add retrieved image RAG | Gemini Pro · MRAG-Bench | accuracy: +4.22 | Helped |
| add ground-truth image RAG | Gemini Pro · MRAG-Bench | accuracy: +9.69 | Helped |
| add retrieved image RAG | Human · MRAG-Bench | accuracy: +22.91 | Helped |
| add ground-truth image RAG | Human · MRAG-Bench | accuracy: +33.16 | Helped |
| use retrieved text instead of retrieved images (ablation) | LLaVA-NeXT-Interleave-7B · MRAG-Bench | accuracy: -5.47 | Hurt |
| use ground-truth text instead of ground-truth images (ablation) | LLaVA-NeXT-Interleave-7B · MRAG-Bench | accuracy: -2.37 | Hurt |
| add both ground-truth images and text rather than ground-truth images alone (ablation) | LLaVA-NeXT-Interleave-7B · MRAG-Bench | accuracy: 47.82 vs 52.99 | Hurt |
| add both ground-truth images and text rather than ground-truth images alone (ablation) | GPT-4-Turbo · MRAG-Bench | accuracy: 62.85 -> 65.11 | Helped |
Surprising findings
- Almost all open-source LVLMs perform worse with retrieved images than with no RAG, while proprietary models improve.
- Humans gain far more from ground-truth RAG than the best LVLM, with GPT-4o gaining only 5.82%.
Limitations
- Benchmark is limited to 1,353 multiple-choice questions across 9 scenarios.
- Retrieval evaluated mainly with CLIP top-5 images; retriever choice strongly affects results.
Experimental setup
Datasets: ImageNet Oxford Flowers102 StanfordCars GeoDE Wikipedia (2023/07/01) Bing Image Search web images
Benchmarks: MRAG-Bench
Metrics: accuracy Recall@5 relative accuracy improvement with RAG
Baselines: GPT-4o GPT-4-Turbo Gemini Pro Claude 3.5 Sonnet OpenFlamingo-v2-9B Idefics2-8B VILA1.5-13B LLaVA-NeXT-Interleave-7B LLaVA-OneVision Mantis-8B mPLUG-Owl3-7B Deepseek-VL-7B-chat
Builds on: OK-VQA A-OKVQA Wiki-LLaVA MagicLens
Closest papers by meaning
- Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation (ICML 2026 · similarity 0.92)
- MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval (ICLR 2026 · similarity 0.91)
- ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.91)
- RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.91)
- Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (ICLR 2025 · similarity 0.91)
- Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.