Research map / Vision-language and multimodal / Visual question answering
Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReflectiVA, a LLaVA-based MLLM that uses special reflective tokens (retrieve/no-retrieve and relevant/irrelevant) trained via a two-stage two-model recipe for knowledge-based VQA.
Problem
MLLMs rely only on parametric knowledge from training, which limits accuracy on knowledge-intensive VQA; they also need to decide when retrieval is needed and which retrieved passages are relevant.
Approach
Extend the MLLM vocabulary with four reflective tokens (<RET>, <NORET>, <REL>, <NOREL>) so the model itself decides whether to retrieve and filters retrieved passages, trained with an in-article discriminator that auto-annotates relevance data for a second-stage overall model.
Key results
- ReflectiVA with EVA-CLIP-8B textual retrieval reaches 40.1 VQA accuracy on InfoSeek validation (All), outperforming EchoSight's 30.4 under the same LLaMA-3.1-8B setup.
- On Encyclopedic-VQA single-hop, ReflectiVA reaches 28.0 (textual) and 35.5 (visual retrieval), versus 22.4 and 26.4 for the reproduced EchoSight.
- With a re-ranking stage, ReflectiVA reaches 40.6 accuracy on Encyclopedic-VQA single-hop, compared to 35.5 without re-ranking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ReflectiVA's reflective retrieval and relevance tokens | LLaMA-3.1-8B · Encyclopedic-VQA test, single-hop | VQA accuracy: 26.4 -> 35.5 | Helped |
| Add ReflectiVA's reflective retrieval and relevance tokens | LLaMA-3.1-8B · InfoSeek validation, all | VQA accuracy: 30.4 -> 40.1 | Helped |
| Add passage re-ranking | ReflectiVA · Encyclopedic-VQA test, single-hop | VQA accuracy: 35.5 -> 40.6 | Helped |
| Add passage re-ranking | ReflectiVA · Encyclopedic-VQA test, all | VQA accuracy: 35.5 -> 39.7 | Helped |
| Use two-stage, two-model training rather than only the in-article reflective model (ablation) | ReflectiVA · Encyclopedic-VQA test, single-hop | VQA accuracy: 21.1 -> 35.5 | Helped |
| Use two-stage, two-model training rather than a single model (ablation) | ReflectiVA · InfoSeek validation, all | VQA accuracy: 28.5 -> 40.1 | Helped |
| Add the ability to decide whether retrieval is needed instead of always using <RET> (ablation) | ReflectiVA · Encyclopedic-VQA test, single-hop | VQA accuracy: 35.3 -> 35.5 | No clear effect |
| Add <REL>/<NOREL> relevance tokens (ablation) | ReflectiVA · Encyclopedic-VQA test, single-hop | VQA accuracy: 23.6 -> 35.5 | Helped |
| Add <REL>/<NOREL> relevance tokens (ablation) | ReflectiVA · InfoSeek validation, all | VQA accuracy: 31.4 -> 40.1 | Helped |
| Use reflective relevance selection instead of Contriever passage selection (ablation) | ReflectiVA · Encyclopedic-VQA test, single-hop | VQA accuracy: 29.3 -> 35.5 | Helped |
| Use reflective relevance selection instead of Contriever passage selection (ablation) | ReflectiVA · InfoSeek validation, all | VQA accuracy: 29.9 -> 40.1 | Helped |
| Add knowledge-base retrieval rather than always using <NORET> (ablation) | ReflectiVA · Encyclopedic-VQA test, single-hop | VQA accuracy: 21.3 -> 35.5 | Helped |
Surprising findings
- Consistently performing retrieval (always <RET>) causes only a minor drop, indicating the retrieval-necessity decision contributes less than relevance filtering.
- Using the top-1 retrieved document is better than top-5 on InfoSeek, while top-10 is best on Encyclopedic-VQA.
Limitations
- Relies on a proprietary LLM (GPT-4) to annotate passage relevance for training data.
- Performance depends on retrieval quality; image-to-image vs. text retrieval optimal settings differ across datasets.
- Slight performance reduction on standard non-knowledge benchmarks relative to the original LLaVA-v1.5.
Experimental setup
Datasets: Encyclopedic-VQA InfoSeek LLaVA-Instruct Wikipedia knowledge bases
Benchmarks: Encyclopedic-VQA InfoSeek MMMU MMBench POPE SEED-Img MME GQA TextVQA ScienceQA AI2D
Metrics: BEM accuracy VQA accuracy relaxed accuracy Recall@k
Baselines: Wiki-LLaVA EchoSight RORA-VLM DPR BLIP-2 InstructBLIP LLaVA-v1.5 GPT-4V
Builds on: LLaVA-v1.5 Wiki-LLaVA EchoSight CLIP Contriever
Closest papers by meaning
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.90)
- CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering (CVPR 2025 · similarity 0.90)
- BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions (AAAI 2024 · similarity 0.90)
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.90)
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.89)
- Notes-guided MLLM Reasoning: Enhancing MLLM with Knowledge and Visual Notes for Visual Question Answering (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.