Research map / Vision-language and multimodal / Multimodal LLMs
Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SpatialEval, a synthetic and real-image benchmark (Spatial-Map, Maze-Nav, Spatial-Grid, Spatial-Real) that evaluates LLMs and VLMs under text-only, vision-only, and vision-text inputs, finding VLMs often underperform their LLM backbones and rely less on images when text is sufficient.
Problem
Spatial understanding and reasoning in LLMs and vision-language models is under-explored; existing VQA benchmarks do not systematically compare text, vision, and combined input modalities for spatial tasks.
Approach
A configurable benchmark with four spatial tasks, each provided in text-only, vision-only, and vision-text formats with redundant information, enabling controlled comparison of LLM vs. VLM backbones and of modality reliance (including noise and random-image ablations).
Key results
- Spatial-Grid: Llama-3 (text-only) reaches 71.9% accuracy and Mistral-7B 62.1%, both well above random guessing, while many VLMs with vision-only input perform near or below chance.
- Vision-text input improves GPT-4V accuracy by 25.6% on Spatial-Grid over vision-only input; GPT-4o reaches 0.989 accuracy on Spatial-Grid with vision-text input.
- On Spatial-Grid, LLaVA-1.6-34B accuracy improves by 20.1% when the image is removed (text-only VTQA ablation), showing VLMs can underuse or be hurt by visual input.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use vision-only input instead of text-only input | VLMs compared with LLMs · Spatial-Map and Spatial-Grid | accuracy: 62.1% -> 47.1% | Hurt |
| Add textual descriptions to vision-only input (ablation) | VLMs · Spatial-Map and Spatial-Grid | accuracy: VQA to VTQA | Helped |
| Remove the original image from vision-text input (ablation) | LLaVA-1.6-34B · Spatial-Grid | accuracy: 20.1% | Helped |
| Replace the original image with a Gaussian noise image (ablation) | LLaVA-1.6-Vicuna-13B · Maze-Nav | accuracy: +6.5% | Helped |
| Replace the original image with a random dataset image (ablation) | VLMs · Maze-Nav | accuracy | Helped |
| Add multimodal training while evaluating with text-only input (ablation) | VLMs compared with their LLM backbones · Spatial reasoning tasks | accuracy | Helped |
| Use text-only LLM input instead of vision-text VLM input (ablation) | LLM and VLM pairs with the same language backbone · Spatial reasoning tasks | accuracy | Mixed |
| Use step-by-step explanation prompting instead of completion prompting (ablation) | Mistral-7B, Vicuna-13B-1.5, Vicuna-7B-1.5, and LLaVA variants · Spatial reasoning tasks, averaged accuracy | accuracy: Vicuna-13B-1.5 text-only: 0.25 -> 0.46; Vicuna-7B-1.5 text-only: 0.31 -> 0.41 | Helped |
| Set decoding temperature to 1 instead of 0.2 (ablation) | Vicuna and LLaVA models · Spatial reasoning tasks, averaged accuracy | accuracy: LLaVA-1.6-Mistral-7B vision-text: 0.46 -> 0.59 | Hurt |
| Set decoding temperature to 1 instead of 0.2 (ablation) | Mistral-7B · Spatial reasoning tasks, averaged accuracy | accuracy: 0.62 -> 0.62 | No clear effect |
| Use vision-text input instead of vision-only input (ablation) | GPT-4V · Spatial-Grid | accuracy: +25.6% | Helped |
| Use vision-only input instead of text-only input | VLMs · Spatial reasoning tasks | accuracy | Hurt |
Surprising findings
- Competitive VLMs with vision-only input can fall behind random guessing on spatial tasks.
- Replacing the original image with a random mismatched image often improves VLM accuracy on Maze-Nav, suggesting weak reliance on visual content when text is sufficient.
Limitations
- Primarily empirical analysis; no new training strategies or architectures proposed.
- Proprietary models evaluated with a single run only due to cost and availability.
- Tasks are synthetic (three of four), limiting ecological validity.
Experimental setup
Datasets: Spatial-Map Maze-Nav Spatial-Grid Spatial-Real Densely Captioned Images
Benchmarks: SpatialEval (Spatial-Map) SpatialEval (Maze-Nav) SpatialEval (Spatial-Grid) SpatialEval (Spatial-Real)
Metrics: accuracy
Baselines: LLaMA-2-7B LLaMA-2-13B LLaMA-3-8B Mistral-7B Vicuna-13B-1.5 LLaVA-1.6-Vicuna-13B GPT-4o GPT-4V Claude 3 Opus Gemini Pro 1.0
Builds on: Densely Captioned Images LLaVA-1.6
Closest papers by meaning
- SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.94)
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.94)
- SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.93)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.93)
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.93)
- Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference under Ambiguities (ICLR 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.