Research map / Vision-language and multimodal / Multimodal LLMs
Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why VLMs fail at simple two-object spatial relations via attention to image tokens, then proposes AdaptVis, a training-free decoding method that sharpens or smooths image attention logits based on generation confidence.
Problem
Large VLMs struggle with basic two-object spatial reasoning (e.g., "under", "behind"), and it is unclear how vision and text tokens interact internally to cause these failures.
Approach
Image tokens receive little attention, and where attention falls spatially matters more than how much. A confidence-gated temperature scaling of image attention logits sharpens attention when the model is confident (familiar relations) and smooths it when confidence is low (unfamiliar relations).
Key results
- AdaptVis raises LLaVA-1.6 Controlled_A accuracy from 48.2 to 98.2 (+50.0 points) on WhatsUp.
- On VSR, AdaptVis improves LLaVA-1.6 exact match from 58.8 to 62.7 and F1 from 29.4 to 39.3.
- On Qwen2-VL, attention intervention improves VG_two_obj accuracy from 56.22 to 66.95 while keeping general benchmarks (POPE, GQA, TextVQA) roughly unchanged or slightly better.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply confidence-guided attention scaling instead of greedy decoding | LLaVA-1.5 · WhatsUp Controlled_A | accuracy: 60.3 -> 84.9 | Helped |
| Apply confidence-guided attention scaling instead of greedy decoding | LLaVA-1.5 · WhatsUp Controlled_B | accuracy: 73.1 -> 83.8 | Helped |
| Apply confidence-guided attention scaling instead of greedy decoding | LLaVA-1.5 · WhatsUp COCO_one | accuracy: 53.0 -> 53.6 | Helped |
| Apply confidence-guided attention scaling instead of greedy decoding | LLaVA-1.5 · WhatsUp COCO_two | accuracy: 58.2 -> 59.9 | Helped |
| Apply confidence-guided attention scaling instead of greedy decoding | LLaVA-1.5 · WhatsUp VG_one | accuracy: 35.9 -> 42.7 | Helped |
| Apply confidence-guided attention scaling instead of greedy decoding | LLaVA-1.5 · WhatsUp VG_two | accuracy: 40.8 -> 48.1 | Helped |
| Apply confidence-guided attention scaling instead of greedy decoding | LLaVA-1.6 · WhatsUp Controlled_A | accuracy: 48.2 -> 98.2 | Helped |
| Apply confidence-guided attention scaling instead of greedy decoding | LLaVA-1.6 · WhatsUp VG_two | accuracy: 7.3 -> 17.2 | Helped |
| Apply confidence-guided attention scaling instead of greedy decoding | LLaVA-1.5 · VSR | F1 Score: 51.3 -> 62.5 | Helped |
| Apply confidence-guided attention scaling instead of greedy decoding | LLaVA-1.6 · VSR | F1 Score: 29.4 -> 39.3 | Helped |
| Intervene in image attention distribution | Qwen2-VL · VSR | accuracy: 78.96 -> 81.60 | Helped |
| Intervene in image attention distribution | Qwen2-VL · VG_two_obj | accuracy: 56.22 -> 66.95 | Helped |
Surprising findings
- Image tokens make up about 90% of the input but receive only about 10% of attention, yet uniformly adding attention to image tokens does not improve accuracy.
- Attention-based overlap with YOLO boxes in middle layers is a strong predictor of answer correctness (high AUROC), while the model's own confidence is a better calibration signal than raw attention.
Limitations
- Optimal alpha and beta vary across distributions and prompts, requiring a validation set for tuning.
- Cannot fix errors originating outside the language model, such as CLIP vision encoder failures.
- Evaluated mainly on LLaVA-1.5/1.6 with a smaller Qwen2-VL study; relies on YOLO annotations for analysis, which have errors.
Experimental setup
Datasets: WhatsUp VSR MS COCO Visual Genome llava_v1_5_mix665k
Benchmarks: WhatsUp VSR POPE GQA TextVQA
Metrics: accuracy pair accuracy set accuracy F1 score AUROC
Baselines: LLaVA-1.5 LLaVA-1.6 VCD DoLa ScalingVis
Builds on: PASTA DoLa VCD
Closest papers by meaning
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.94)
- Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models (ICLR 2026 · similarity 0.93)
- Visual Structures Help Visual Reasoning: Addressing the Binding Problem in LVLMs (NeurIPS 2025 · similarity 0.93)
- SIFThinker: Spatially-Aware Image Focus for Visual Reasoning (AAAI 2026 · similarity 0.92)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.