Research map / Vision-language and multimodal / Visual question answering
Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReAuSE, which turns a generative MLLM (MiniGPT-4-v2-7B with LoRA) into a built-in autoregressive search engine that generates FM-index-constrained document identifiers for retrieval-augmented knowledge-based VQA, further calibrated by DPO on relevance-feedback rewards.
Problem
Knowledge-based VQA with retrieval-augmented generation relies on separate discriminative retrievers and generators, which are heterogeneous and hard to jointly optimize, and retrieval degrades on large knowledge bases.
Approach
Integrate retrieval into the generative MLLM itself: the model generates substring document identifiers that are constrained to appear in the knowledge base (via FM-Index), and its retrieval is refined with DPO using VQA-score, keyword-hit, and similarity rewards from relevance feedback.
Key results
- On OKVQA, ReAuSE reaches 65.7 VQA score versus 62.1 for FLMR and 64.3 for GPT-4V.
- On A-OKVQA direct-answer test, ReAuSE reaches 65.8 versus 60.1 for CKR-VQA; multi-choice test 80.3 versus 76.7 for Cola-FT.
- On OKVQA-Wiki21M retrieval, PRRecall@5 is 88.0 versus 68.1 for FLMR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReAuSE instead of KSVQA | OKVQA · OKVQA | Score: 62.8 -> 65.7 | Helped |
| Use ReAuSE instead of FLMR | OKVQA · OKVQA | PRR@K: 89.3 -> 92.6 | Helped |
| Use ReAuSE instead of strongest listed baseline | A-OKVQA multi-choice validation · A-OKVQA multi-choice val | accuracy: 78.1 -> 85.0 | Helped |
| Use ReAuSE instead of strongest listed baseline | A-OKVQA multi-choice test · A-OKVQA multi-choice test | accuracy: 76.7 -> 80.3 | Helped |
| Use ReAuSE instead of strongest listed baseline | A-OKVQA direct-answer validation · A-OKVQA direct-answer val | VQA score: 63.7 -> 67.7 | Helped |
| Use ReAuSE instead of strongest listed baseline | A-OKVQA direct-answer test · A-OKVQA direct-answer test | VQA score: 60.1 -> 65.8 | Helped |
| Add built-in autoregressive search engine (ablation) | ReAuSE · OKVQA | Score: 61.9 -> 65.7 | Helped |
| Add supervised fine-tuning of the search engine (ablation) | ReAuSE · OKVQA | Score: 61.6 -> 65.7 | Helped |
| Add constrained decoding (ablation) | ReAuSE · OKVQA | Score: 63.2 -> 65.7 | Helped |
| Add retrieval calibration (ablation) | ReAuSE · OKVQA | PRRecall@5: 88.7 -> 92.6 | Helped |
| Add the VQA reward model (ablation) | ReAuSE retrieval calibration · OKVQA | Score: 63.3 -> 65.7 | Helped |
| Add the EM reward function (ablation) | ReAuSE retrieval calibration · OKVQA | PRRecall@5: 89.9 -> 92.6 | Helped |
Surprising findings
- Removing constrained decoding costs only about 2.5 VQA points while removing the whole search engine costs 3.8 points, and the paper reports the unconstrained generated knowledge is hallucinated.
Limitations
- Retrieval on the Wiki21M corpus and InfoSeek is weaker than on OKVQA GS112K gains, and retrieval time grows with TopK.
- Relies on an MLLM reward model and LLM-based extractive summarizer to build identifiers and rewards, adding pipeline complexity.
- Results on OKVQA are compared against baselines of differing scales and some baselines were reproduced by the authors.
Experimental setup
Datasets: OKVQA A-OKVQA GS112K Wiki21M InfoSeek PreFLMR KB
Benchmarks: OKVQA A-OKVQA InfoSeek
Metrics: VQA score accuracy PRRecall@5 PRRecall@10
Baselines: LLaVA-13B PaLM-E-562B GPT-4V Prophet PromptCap FillingGap REVIVE FLMR GeMKR ReVeaL TwO Cola-FT
Builds on: DSI FLMR DPO LoRA RA-VQA
Closest papers by meaning
- Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering (NeurIPS 2025 · similarity 0.91)
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.91)
- VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents (CVPR 2025 · similarity 0.91)
- Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (ICLR 2025 · similarity 0.90)
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.90)
- Generative Multi-Modal Knowledge Retrieval with Large Language Models (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.