Research map / Vision-language and multimodal / Visual question answering
MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows MLLMs are sensitive to visual subject size, that they attend to the correct region even when wrong, and proposes training-free attention/gradient-based visual cropping (ViCrop) that improves small-detail VQA accuracy.
Problem
MLLMs perform worse on questions about small visual details; it is unclear whether this stems from failing to locate the subject or failing to perceive its details.
Approach
MLLMs' internal answer-to-image attention (normalized by a generic instruction, or weighted by gradients) localizes the question's subject even when the answer is wrong; this map is used to choose a crop that is concatenated with the full image tokens, improving perception without training.
Key results
- On TextVQA, LLaVA-1.5 accuracy rises from 47.80 (no cropping) to 56.06 with grad-att and 55.17 with rel-att; InstructBLIP rises from 33.48 to 45.71 with grad-att.
- On V*, LLaVA-1.5 rel-att improves accuracy from 42.41 to 62.30.
- Attention ratio over TextVQA is greater than 1 in most layers for all studied MLLMs, including on incorrectly answered questions, indicating MLLMs know where to look.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply relative-attention, gradient-weighted attention, or input-gradient cropping | LLaVA-1.5 · TextVQA, V*, POPE, DocVQA | VQA score: TextVQA: 47.80 -> 55.17/56.06; V*: 42.41 -> 62.30/57.07/46.07; POPE: 85.27 -> 87.25/87.03/86.06; DocVQA: 15.97 -> 19.63/ | Helped |
| Apply ViCrop methods | InstructBLIP · TextVQA, V*, POPE, DocVQA | VQA score: TextVQA: 33.48 -> 45.44/45.71/42.23; V*: 35.60 -> 42.41/37.70/37.17; POPE: 84.89 -> 86.64/86.99/86.84; DocVQA: 9.20 -> 9 | Helped |
| Apply ViCrop methods | LLaVA-1.5 and InstructBLIP · AOKVQA, GQA, VQAv2 | accuracy / VQA score: LLaVA-1.5: AOKVQA 59.01 -> 60.66/59.94/59.92; GQA 60.48 -> 60.97/60.98/60.54; VQAv2 75.57 -> 76.51/76.06/75.94. Instruct | Mixed |
| Use human-provided cropping instead of no cropping | MLLMs on TextVQA · TextVQA size partitions | accuracy: BLIP-2 small: 12.13 -> 55.76; medium: 19.57 -> 52.02; large: 36.32 -> 45.73; InstructBLIP small: 21.79 -> 69.60; medium: | Helped |
| Average all layers instead of selecting a target layer for rel-att (ablation) | LLaVA-1.5 · TextVQA | VQA score: +0.28 | Helped |
| Average all layers instead of selecting a target layer for grad-att (ablation) | LLaVA-1.5 · TextVQA | VQA score: +0.20 | Helped |
| Average all layers instead of selecting a target layer for rel-att (ablation) | InstructBLIP · TextVQA | VQA score: -1.04 | Hurt |
| Average all layers instead of selecting a target layer for grad-att (ablation) | InstructBLIP · TextVQA | VQA score: -0.73 | Hurt |
| Use high-resolution two-stage cropping (ablation) | LLaVA-1.5 ViCrop · V* | VQA score: rel-att: 47.64 -> 62.30; grad-att: 49.74 -> 57.07; pure-grad: 45.03 -> 46.07 | Helped |
| Use high-resolution two-stage cropping (ablation) | InstructBLIP grad-att and pure-grad · V* | VQA score: grad-att: 42.41 -> 37.70; pure-grad: 42.41 -> 37.17 | Hurt |
| Use relative-attention cropping rather than external localization tools | LLaVA-1.5 and InstructBLIP · TextVQA | VQA score: LLaVA-1.5: SAM 49.42, YOLO 48.84, CLIP 48.55, rel-att 55.17; InstructBLIP: SAM 39.23, YOLO 36.49, CLIP 39.61, rel-att 45 | Helped |
| Apply rel-att cropping | LLaVA-NeXT (Mistral-7B) · TextVQA and V* | VQA score: TextVQA: 65.17 -> 68.65; V*: 58.11 -> 61.78 | Helped |
Surprising findings
- MLLMs attend to the correct answer region even when they answer incorrectly, suggesting the failure is perception rather than localization.
- The cropping methods hurt InstructBLIP on V* at high resolution (grad-att and pure-grad decline with the high-res strategy), and InstructBLIP gains less than LLaVA-1.5 because its LLM is not tuned to use the extra image tokens.
Limitations
- Gains are smaller for relation and counting questions since ViCrop focuses on only one region.
- Adds inference-time overhead and extra visual tokens.
- Manual bounding-box annotation was needed for the TextVQA analysis, with a filtered set of 4370 pairs.
Experimental setup
Datasets: TextVQA V* POPE DocVQA AOKVQA GQA VQAv2
Benchmarks: TextVQA V* POPE DocVQA AOKVQA GQA VQAv2
Metrics: VQA score accuracy attention ratio inference time
Baselines: SEAL SAM ViCrop YOLO ViCrop CLIP ViCrop LLaVA-NeXT human-CROP
Builds on: Grad-CAM LLaVA-1.5 InstructBLIP
Closest papers by meaning
- FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering (NeurIPS 2025 · similarity 0.90)
- Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly (CVPR 2025 · similarity 0.90)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.90)
- Visual Perception by Large Language Model’s Weights (NeurIPS 2024 · similarity 0.90)
- Notes-guided MLLM Reasoning: Enhancing MLLM with Knowledge and Visual Notes for Visual Question Answering (CVPR 2025 · similarity 0.90)
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.