Research map / Vision-language and multimodal / Multimodal LLMs
RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RAG-IGBench, a 6,057-sample benchmark for RAG-based interleaved image-text generation in open-domain QA, with rule-based text, image, and image-text consistency metrics, evaluated across proprietary and open-source MLLMs.
Problem
Evaluating interleaved image-text responses for open-domain questions is hard because existing benchmarks rely on unimodal metrics or MLLM-based judges that are biased and unstable.
Approach
Frame interleaved generation as RAG-based image selection and placement from retrieved image-text documents, and score outputs with rule-based metrics against ground-truth answers: ROUGE-1 for text, edit distance and Kendall score for image selection and order, and a context-based Alignment Score plus CLIP score for image-text consistency.
Key results
- GPT-4o achieves the top mean score of 46.01 on RAG-IGBench among evaluated models, and Qwen2.5VL-72B (37.79) is the best open-source model.
- Rule-based metrics correlate better with human judgments than GPT-4o-based evaluation on 200 sampled cases (e.g., image-text consistency Pearson 0.172 vs 0.151; overall Pearson 0.143 vs 0.017).
- SFT on the RAG-IGBench training set raises Qwen2VL-7B's RAG-IGBench dev score from 26.12 to 36.10 and improves average across listed benchmarks by +3.90.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2VL-7B on the RAG-IGBench training set | Qwen2VL-7B · RAG-IGBench (Dev) | Mean: 26.12 -> 36.10 (+9.98) | Helped |
| Fine-tune Qwen2VL-7B on the RAG-IGBench training set | Qwen2VL-7B · MMB | Score: 81.62 -> 82.04 (+0.42) | Helped |
| Fine-tune Qwen2VL-7B on the RAG-IGBench training set | Qwen2VL-7B · BLINK | Score: 53.34 -> 54.50 (+1.16) | Helped |
| Fine-tune Qwen2VL-7B on the RAG-IGBench training set | Qwen2VL-7B · Mantis-Eval | Score: 49.06 -> 57.55 (+8.49) | Helped |
| Fine-tune Qwen2VL-7B on the RAG-IGBench training set | Qwen2VL-7B · Q-Bench2 | Score: 61.80 -> 68.60 (+6.80) | Helped |
| Fine-tune Qwen2VL-7B on the RAG-IGBench training set | Qwen2VL-7B · NLVR2 | Score: 84.01 -> 80.56 (-3.45) | Hurt |
| Add SFT (ablation) | Qwen2.5VL-3B-Instruct · RAG-IGBench | Mean: 16.30 -> 33.11 | Helped |
| Add GRPO (ablation) | Qwen2.5VL-3B-Instruct · RAG-IGBench | Mean: 16.30 -> 31.63 | Helped |
| Add both SFT and GRPO (ablation) | Qwen2.5VL-3B-Instruct · RAG-IGBench | Mean: 16.30 -> 33.31 | Helped |
| Add SFT (ablation) | Qwen2.5VL-7B-Instruct · RAG-IGBench | Mean: 22.89 -> 33.77 | Helped |
| Add GRPO (ablation) | Qwen2.5VL-7B-Instruct · RAG-IGBench | Mean: 22.89 -> 33.00 | Helped |
| Add both SFT and GRPO (ablation) | Qwen2.5VL-7B-Instruct · RAG-IGBench | Mean: 22.89 -> 33.81 | Helped |
Surprising findings
- GRPO with the proposed metrics as rewards yields smaller gains on text quality than SFT and mainly improves image-related and consistency metrics.
- Adding more input images degrades performance even at low resolution, and performance peaks at moderate resolution with few images.
Limitations
- Data comes exclusively from Xiaohongshu, a single Chinese social platform, and the corpus is predominantly Chinese, translated with GPT-4o.
- Benchmark scale (6,057 samples) is limited, restricting pre-training use.
- Ground-truth answers are partly generated by MLLMs (GPT-4o, Claude, Gemini), which may introduce bias despite manual refinement.
Experimental setup
Datasets: RAG-IGBench MMB BLINK Mantis-Eval Q-Bench2 NLVR2
Benchmarks: RAG-IGBench
Metrics: ROUGE-1 edit distance Kendall score alignment score CLIP score Pearson correlation Spearman correlation
Baselines: GPT-4o Claude-3.5-Sonnet Gemini-1.5-pro QwenVL-Max Qwen2VL-72B InternVL2.5 78B Qwen2.5VL-72B InterleavedBench MMIE
Builds on: retrieval-augmented generation CLIP ROUGE GRPO VLM-R1
Closest papers by meaning
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.90)
- VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents (CVPR 2025 · similarity 0.89)
- Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines (AAAI 2025 · similarity 0.89)
- Benchmarking Large Language Models in Retrieval-Augmented Generation (AAAI 2024 · similarity 0.89)
- Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution (AAAI 2025 · similarity 0.89)
- Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.