Research map / Vision-language and multimodal / Visual question answering
Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Wiki-PRF, a three-stage processing-retrieval-filtering multimodal RAG framework where a GRPO-trained Qwen2.5-VL (VLM-PRF) invokes visual tools, retrieves Wikipedia sections via EVA-CLIP, and filters them, reaching SOTA on E-VQA and InfoSeek.
Problem
Knowledge-based VQA requires retrieving external knowledge, but RAG methods struggle with low-quality multimodal queries (fine-grained regions lost to dominant objects) and with filtering noisy retrieved passages.
Approach
A VLM is RL-trained (GRPO, LoRA, no KL) with answer-accuracy and format rewards to decide which visual tools (captioning, grounding, flipping) to invoke for retrieval queries and to filter retrieved sections question-specifically, replacing passage-level reranking.
Key results
- VLM-PRF-7B reaches 36.0 on E-VQA and 42.8 on InfoSeek, reported as SOTA over ReflectiVA and MMKB-RAG.
- Wiki-PRF-7B achieves 77.8 on OK-VQA, versus 73.1 for KU-RAG.
- RL outperforms SFT on a 2,000-instance InfoSeek sample (46.3 vs 41.8 overall).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RL fine-tuning instead of leaving Wiki-PRF without RL | Wiki-PRF-7B · InfoSeek All | VQA accuracy: 39.5 -> 42.8 | Helped |
| Add processing to retrieval (ablation) | VLM-PRF-3B · InfoSeek validation, 10K samples | VQA accuracy: 34.22 -> 36.76 | Helped |
| Add filtering to retrieval (ablation) | VLM-PRF-3B · InfoSeek validation, 10K samples | VQA accuracy: 34.22 -> 36.24 | Helped |
| Add multi-tool processing and filtering to retrieval (ablation) | VLM-PRF-3B · InfoSeek validation, 10K samples | VQA accuracy: 34.22 -> 39.48 | Helped |
| Use captioning rather than grounding as the sole tool (ablation) | VLM-PRF-3B with all stages · InfoSeek validation, 10K samples | VQA accuracy: 37.22 -> 38.18 | Helped |
| Add captioning and grounding tools together rather than one alone (ablation) | VLM-PRF-3B with all stages · InfoSeek validation, 10K samples | VQA accuracy: 38.18 -> 39.48 | Helped |
| Increase training sample size from 2K to 8K (ablation) | VLM-PRF-3B · InfoSeek | VQA accuracy: 37.30% -> 40.83% | Helped |
| Increase retrieved article count from Top-1 to Top-7 (ablation) | Wiki-PRF-3B · InfoSeek | VQA accuracy: 38.85% -> 39.57% | Helped |
| Increase knowledge-base size from 10k to 100k entries (ablation) | Wiki-PRF-7B · InfoSeek | VQA accuracy: 60.3 -> 42.8 | Hurt |
| Use equal answer-reward and format-reward weighting rather than the other tested ratios (ablation) | VLM-PRF-3B · InfoSeek | VQA accuracy: 38.80, 38.20, 39.48, 38.89, 38.53 | Helped |
| Use Top-3 rather than Top-1 retrieved sections (ablation) | Wiki-PRF-3B · E-VQA | VQA accuracy: 24.28% -> 24.31%; 28.15% -> 28.94%; 32.10% -> 32.38% | Mixed |
| Use RL rather than SFT for filtering | Wiki-PRF-7B · InfoSeek (2,000 sampled instances), All | VQA accuracy: 41.8 -> 46.3 | Helped |
Surprising findings
- Table 1 reports InternVL3-8B variant with higher E-VQA accuracy (39.2) than the 7B Qwen variant, and the 3B/7B models differ in tool-usage combinations in unexpected ways.
- Results on E-VQA for the best system are reported as 36.0 while MMKB-RAG leads on single-hop E-VQA (39.7).
Limitations
- Limited to three visual tools (captioning, grounding, flipping) in this study.
- Processing and filtering stages add substantial inference latency from tool calls and long-text processing.
Experimental setup
Datasets: InfoSeek E-VQA OK-VQA OVEN
Benchmarks: E-VQA InfoSeek OK-VQA
Metrics: VQA accuracy BEM score recall@k
Baselines: BLIP-2 InstructBLIP LLaVA-v1.5 GPT-4V DPR RORA-VLM EchoSight Wiki-LLaVA ReflectiVA MMKB-RAG KU-RAG
Builds on: GRPO VLM-R1 Visual-RFT EchoSight
Closest papers by meaning
- Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines (AAAI 2025 · similarity 0.91)
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.90)
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.90)
- Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (ICLR 2025 · similarity 0.90)
- ConVQG: Contrastive Visual Question Generation with Multimodal Guidance (AAAI 2024 · similarity 0.89)
- SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.