Research map / Vision-language and multimodal / Visual question answering
Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Dyn-VQA, a 1,452-question bilingual benchmark of dynamic multimodal questions, and OmniSearch, a self-adaptive planning agent that decomposes questions into retrieval sub-question chains, outperforming heuristic mRAG baselines.
Problem
Existing heuristic multimodal RAG pipelines use fixed, single-pass retrieval, and current knowledge-seeking VQA benchmarks are too shallow (mostly two-hop, static textual knowledge) to expose their rigidity in query, tool, and timing of retrieval.
Approach
A new benchmark of dynamic questions (fast-changing answers, multimodal knowledge needs, multi-hop) plus an agent that plans each retrieval action (sub-question, retrieval API, query) adaptively from retrieved feedback, implemented either by prompting GPT-4V or by LoRA fine-tuning Qwen-VL-Chat on synthesized planning trajectories.
Key results
- OmniSearch (GPT-4V) reaches 50.03 overall F1-Recall on Dyn-VQA versus 36.10 for GPT-4V with two-step image-caption-then-web mRAG.
- OmniSearch (Qwen-VL-Chat), LoRA fine-tuned, reaches 41.20 overall, surpassing GPT-4V with two-step mRAG (36.10) and Qwen-VL-Chat with the same pipeline (23.16 for the image-first variant).
- Human performance on Dyn-VQA is 55.12, while the best generative search engine (Gemini) reaches 37.69.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive multi-step retrieval planning instead of image-caption two-step heuristic retrieval | GPT-4V on Dyn-VQA · Dyn-VQA | F1-Recall: 37.15 -> 50.03 | Helped |
| use GPT-4V as sub-question solver instead of OmniSearch (Q) (ablation) | OmniSearch (Q) on Dyn-VQA · Dyn-VQA | F1-Recall: 41.20 -> 43.15 | Helped |
| use GPT-4V for multimodal and GPT-4 for text-only sub-questions (ablation) | OmniSearch (Q) on Dyn-VQA · Dyn-VQA | F1-Recall: 43.15 -> 44.89 | Helped |
| replace OmniSearch solver with Qwen-VL-Chat (ablation) | OmniSearch (Q) on Dyn-VQA · Dyn-VQA | F1-Recall: 41.20 -> 40.17 | Hurt |
| replace OmniSearch solver with Qwen-VL-Chat (ablation) | OmniSearch (G) on Dyn-VQA · Dyn-VQA | F1-Recall: 50.03 -> 46.07 | Hurt |
| add retrieved image captions (ablation) | OmniSearch (G) on Dyn-VQA · Dyn-VQA | F1-Recall: 47.64 -> 50.03 | Helped |
| add retrieved images (ablation) | OmniSearch (G) on Dyn-VQA · Dyn-VQA | F1-Recall: 48.57 -> 50.03 | Helped |
| add related knowledge to web title and description (ablation) | OmniSearch (G) on Dyn-VQA · Dyn-VQA | F1-Recall: 50.03 -> 51.25 | Helped |
| include web title in retrieved snippets (ablation) | OmniSearch (G) on Dyn-VQA · Dyn-VQA | F1-Recall: 48.08 -> 50.03 | Helped |
| use all retrieved contents instead of 3 (ablation) | OmniSearch (G) on Dyn-VQA · Dyn-VQA | F1-Recall: 48.09 -> 50.03 | Helped |
| use all retrieved contents instead of 3 (ablation) | GPT-4V on Dyn-VQA · Dyn-VQA | F1-Recall: 38.78 -> 37.15 | Hurt |
Surprising findings
- Generative search engines (Bing Chat, Perplexity AI, Gemini) perform poorly on Dyn-VQA, failing to ground pronouns like 'it' to image objects.
- Image caption retrieval helps on dynamic questions but its advantage reverses for questions not requiring external visual knowledge.
Limitations
- Dyn-VQA is small (1,452 questions) and the transportation domain has only 10 instances.
- A large gap remains between OmniSearch and human performance on fast-changing, multi-hop, and visual-knowledge questions.
- GPT-4V-based OmniSearch tends to over-retrieve, sometimes obscuring correct answers.
Experimental setup
Datasets: InfoSeek CogVLM-SFT-311K FreshQA
Benchmarks: Dyn-VQA
Metrics: F1-Recall human performance comparison
Baselines: Heuristic mRAG with image retrieval Heuristic mRAG with web search Two-step mRAG (image caption first) Bing Chat Perplexity AI Gemini Qwen-VL-Max GPT-4V
Builds on: Chain-of-Thought InfoSeek FreshQA Qwen-VL
Closest papers by meaning
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.91)
- MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.91)
- Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines (AAAI 2025 · similarity 0.90)
- Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering (NeurIPS 2025 · similarity 0.90)
- Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering (NeurIPS 2025 · similarity 0.90)
- Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.