Research map / Vision-language and multimodal / Visual question answering
Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMDocRAG, a benchmark of 4,055 expert-annotated document QA pairs with multimodal evidence chains and interleaved text-image answers, plus quote-selection F1 and LLM-judge metrics, evaluating 60 VLM/LLMs and 14 retrievers.
Problem
Existing DocVQA/DocRAG benchmarks evaluate only retrieval recall or text answers and lack ways to assess selection of relevant multimodal evidence from noisy quotes and integration of images with text in answers.
Approach
A benchmark where models receive 15 or 20 candidate quotes (gold mixed with hard text and image negatives) from long multi-page documents and must select gold quotes and produce interleaved text-image answers, scored with quote-selection F1, BLEU/ROUGE-L, and five LLM-judged dimensions.
Key results
- GPT-4.1 achieves the best quote selection F1 of 70.2 with 20 quotes, and an average answer quality of 4.14.
- Fine-tuning Qwen2.5-32B-Instruct raises quote selection F1 from 58.9 to 65.1 in the pure-text setting.
- End-to-end RAG with single BGE retriever drops GPT-4.1 quote F1 from 70.2 to 54.4 and answer quality from 4.14 to 3.53.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune Qwen2.5-32B-Inst on MMDocRAG dev set (ablation) | Qwen2.5-32B-Inst with 20 pure-text quotes · MMDocRAG | quote selection F1: 58.9 -> 65.1 | Helped |
| fine-tune Qwen2.5-32B-Inst on MMDocRAG dev set (ablation) | Qwen2.5-32B-Inst with 20 pure-text quotes · MMDocRAG | answer quality average: 3.63 -> 3.93 | Helped |
| fine-tune Qwen2.5-VL-7B-Inst on MMDocRAG dev set (ablation) | Qwen2.5-VL-7B-Inst with 20 multimodal quotes · MMDocRAG | quote selection F1: 16.6 -> 58.6 | Helped |
| fine-tune InternVL3-8B on MMDocRAG dev set (ablation) | InternVL3-8B with 20 multimodal quotes · MMDocRAG | quote selection F1: 37.0 -> 58.8 | Helped |
| use multimodal rather than pure-text quotes | GPT-4.1 · MMDocRAG, 20 quotes | quote selection F1: 68.3 -> 70.2 | Helped |
| use multimodal rather than pure-text quotes | Qwen2.5-7B versus Qwen2.5-VL-7B · MMDocRAG, 20 quotes | quote selection F1: 45.8 vs 16.6 | Hurt |
| expand BGE query into clauses | GPT-4.1 end-to-end RAG · MMDocRAG | quote selection F1: 54.4 -> 57.5 | Helped |
| use a multi-retriever ensemble with clause queries instead of single BGE original query | GPT-4.1 end-to-end RAG · MMDocRAG | quote selection F1: 54.4 -> 59.9 | Helped |
| replace perfect retrieval with BGE using original queries | GPT-4.1 end-to-end RAG · MMDocRAG | answer quality: 4.14 -> 3.53 (-14.7%) | Hurt |
| use VLM-generated image descriptions rather than OCR-extracted text | LLMs using text representations of image quotes · MMDocRAG | image quote selection and answer generation | Helped |
| use thinking models for multimodal quote selection and integration | Thinking models · MMDocRAG | answer quality | No clear effect |
| use multiple-page or multiple-image evidence rather than single-image/page evidence | Models evaluated on MMDocRAG · MMDocRAG evidence configurations | model performance | Hurt |
Surprising findings
- Thinking models do not outperform non-thinking models despite about 3x more output tokens.
- Smaller open-source VLMs perform worse with multimodal inputs than with pure-text inputs; e.g., Qwen-7B and 32B LLMs score 175.9% and 62.7% higher F1 than their VLM counterparts.
Limitations
- Quote selection and answer evaluation rely on LLM-as-judge scoring, which may be biased.
- Evaluation uses a fixed candidate quote set, isolating generation from retrieval errors.
- Smaller VLMs struggle with long multimodal input sequences.
Experimental setup
Datasets: MMDocIR MMDocRAG dev-set
Benchmarks: MMDocRAG
Metrics: quote selection F1 BLEU ROUGE-L LLM-as-judge score (fluency, citation quality, text-image co retrieval recall
Baselines: MP-DocVQA DUDE SlideVQA MMLongBench-Doc DocBench M3DocVQA M-Longdoc MMDocIR MuRAR M2RAG
Builds on: MMDocIR MinerU LayoutLMv3 BGE
Closest papers by meaning
- MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.91)
- TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning (NeurIPS 2026 · similarity 0.90)
- Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (ICLR 2025 · similarity 0.90)
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.90)
- Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.90)
- Look as You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.