Research map / Vision-language and multimodal / Visual question answering
Notes-guided MLLM Reasoning: Enhancing MLLM with Knowledge and Visual Notes for Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NoteMR, a training-free KB-VQA framework that uses a frozen MLLM to turn retrieved explicit knowledge into knowledge notes, GradCAM-based visual notes, and candidate-based answer selection.
Problem
Knowledge-based VQA where retrieved explicit knowledge is noisy and MLLMs fail to retrieve relevant implicit knowledge or perceive fine-grained visual details, causing hallucinations.
Approach
Use retrieved passages plus the image as prompts so a frozen MLLM summarizes them into knowledge notes that combine explicit and implicit knowledge; then compute GradCAM cross-modal attention between image patches and the notes to build visual notes that mask irrelevant regions; finally select the best answer among candidates.
Key results
- NoteMR with LLaVA-NeXT-8B reaches 70.0% on OK-VQA versus 62.2% default and 63.3% for SKP, and 68.2% with LLaVA-NeXT-7B.
- On A-OKVQA direct answer, NoteMR scores 68.7% vs 65.3% for SKP; on multiple choice it scores 88.1% vs 76.4% for Prophet.
- Ablation on OK-VQA: raw retrieved knowledge gives 65.3%, knowledge notes 68.8%, adding visual notes 69.6%, and candidate outputs 70.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add NoteMR to LLaVA-NeXT-8B | LLaVA-NeXT-8B · OK-VQA | Score: 62.2 -> 70.0 | Helped |
| add NoteMR to Qwen2-VL-7b | Qwen2-VL-7b · OK-VQA | Score: +6.1% | Helped |
| add NoteMR | LLaVA-NeXT-8B · A-OKVQA MC | Accuracy: 11.7% | Helped |
| add NoteMR | LLaVA-NeXT-8B · A-OKVQA DA | Score: 3.4% | Helped |
| add raw retrieved knowledge (ablation) | LLaVA-NeXT-8B · OK-VQA | Score: 62.2 -> 65.3 | Helped |
| replace raw retrieved knowledge with knowledge notes (ablation) | LLaVA-NeXT-8B with retrieved knowledge · OK-VQA | Score: 65.3 -> 68.8 | Helped |
| add visual notes (ablation) | LLaVA-NeXT-8B with knowledge notes · OK-VQA | Score: 68.8 -> 69.6 | Helped |
| add candidate outputs (ablation) | LLaVA-NeXT-8B with knowledge notes and visual notes · OK-VQA | Score: 69.6 -> 70.0 | Helped |
Surprising findings
- Feeding raw retrieved knowledge to the MLLM lowers accuracy relative to the MLLM's own implicit knowledge in some cases, and knowledge notes outperform raw knowledge by 3.5 points.
Limitations
- Relies on frozen MLLM with no fine-tuning, so gains depend on backbone knowledge.
- Evaluated only on OK-VQA and A-OKVQA with validation sets for A-OKVQA.
Experimental setup
Datasets: OK-VQA A-OKVQA Google Search Corpus Wikipedia
Benchmarks: OK-VQA A-OKVQA
Metrics: VQA score multiple-choice accuracy
Baselines: RA-VQA-v2 TRiG KAT REVIVE Flamingo PromptCap Prophet ASB SKP PaLM-E LLaVA-13B Qwen2-VL-7b
Builds on: PREFLMR GradCAM Prophet SKP LLaVA-NeXT
Closest papers by meaning
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.90)
- Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.89)
- Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.89)
- VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning (ICLR 2025 · similarity 0.89)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.89)
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.