Research map / Vision-language and multimodal / Visual question answering
Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RCTS, a training-free multimodal RAG framework that builds a reasoning-context-enriched VQA knowledge base and re-ranks retrieved in-context examples with MCTS using self-consistency and mutual heuristic rewards.
Problem
LVLMs on VQA suffer from instruction misalignment and erratic responses; existing multimodal RAG retrieves rigid question-answer pairs without reasoning and does not select the most useful examples for in-context learning.
Approach
Auto-generate step-by-step reasoning contexts for VQA pairs, validated by self-consistent answer prediction, and re-rank the top retrieved examples via Monte Carlo Tree Search whose reward combines self-consistency and mutual (cross-example) heuristic rewards.
Key results
- RCTS with Qwen2-VL-2B reaches 78.99% on ScienceQA, vs 67.18% zero-shot and 71.94% Vanilla-RAG.
- RCTS with Qwen2-VL-7B reaches 28.95% on MathV testmini, vs 24.67% Vanilla-RAG.
- On VizWiz and VSR-MC with Qwen2-VL-7B, RCTS gets 71.50% and 55.97%, improving on Vanilla-RAG by 1.61 and 3.05 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RCTS instead of Vanilla-RAG | Qwen2-VL-7B · ScienceQA test | accuracy: 86.68 -> 91.44 | Helped |
| Use RCTS instead of Vanilla-RAG | Qwen2-VL-7B · MMMU-Dev | accuracy: 50.00 -> 53.33 | Helped |
| Use RCTS instead of Vanilla-RAG | Qwen2-VL-7B · MathV testmini | accuracy: 24.67 -> 28.95 | Helped |
| Use RCTS instead of Vanilla-RAG | Qwen2-VL-7B · VizWiz val | accuracy: +1.61% | Helped |
| Use RCTS instead of Vanilla-RAG | Qwen2-VL-7B · VSR-MC test | accuracy: +3.05% | Helped |
| Add reasoning context to MCTS (ablation) | RCTS, Qwen2-VL-7B · ScienceQA | accuracy: 88.92 -> 91.44 | Helped |
| Add reasoning context to MCTS (ablation) | RCTS, Qwen2-VL-7B · MMMU | accuracy: 49.33 -> 53.33 | Helped |
| Add reasoning context to MCTS (ablation) | RCTS, Qwen2-VL-7B · Math-V | accuracy: 25.65 -> 28.95 | Helped |
| Add MCTS to reasoning-context retrieval (ablation) | RCTS, Qwen2-VL-7B · ScienceQA | accuracy: 88.33 -> 91.44 | Helped |
| Add MCTS to reasoning-context retrieval (ablation) | RCTS, Qwen2-VL-7B · MMMU | accuracy: 50.60 -> 53.33 | Helped |
| Add MCTS to reasoning-context retrieval (ablation) | RCTS, Qwen2-VL-7B · Math-V | accuracy: 26.97 -> 28.95 | Helped |
| Set reward weight alpha to 0.2 rather than other tested weights (ablation) | RCTS, Qwen2-VL-7B · ScienceQA / MMMU / Math-V | accuracy | Mixed |
Surprising findings
- Random-retrieval ICL underperforms zero-shot on MMMU and MathV for several models.
- Vanilla-RAG with top retrieval is worse than zero-shot for InternVL-2 on MMMU and MathV.
Limitations
- Performance depends on whether helpful samples exist in the knowledge base.
- Higher computational overhead from MCTS rollouts and multiple generations per candidate.
Experimental setup
Datasets: ScienceQA MMMU MathV VizWiz VSR-MC
Benchmarks: ScienceQA MMMU MathV VizWiz VSR-MC
Metrics: accuracy
Baselines: Zero-Shot ICL (random retrieval) Vanilla-RAG (top retrieval)
Builds on: Auto-CoT Monte Carlo Tree Search PreFLMR Vanilla-RAG in-context learning
Closest papers by meaning
- SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge (NeurIPS 2024 · similarity 0.92)
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.91)
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.91)
- Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.90)
- MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.90)
- Question Aware Vision Transformer for Multimodal Reasoning (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.