Research map / Vision-language and multimodal / Multimodal LLMs
Understanding VLMs Spatial Mental Modeling Capability from Limited Views
Qineng Wang, Baiqiao Yin, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, Saining Xie, Jiajun Wu and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MindCube, a benchmark of 21,154 questions over 3,268 images testing whether VLMs can form spatial mental models from limited views, and finds existing VLMs near random.
Problem
Vision language models may fail to build internal spatial mental models of unseen space from only a few views, which humans do for reasoning about layout, perspective, and motion.
Approach
A new benchmark (MindCube) with 21,154 questions over 3,268 multi-view images to systematically probe spatial mental modeling from limited views; existing VLMs perform near random on it.
Key results
- Existing VLMs exhibit near-random performance on the MindCube benchmark of 21,154 questions across 3,268 images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| insert interpolated views between the given views (ablation) | frozen Qwen2.5-VL-3B · MindCube-Tiny | accuracy: 37.81 -> 37.90 | No clear effect |
| supply a ground-truth augmented cognitive map as input without reasoning (ablation) | frozen Qwen2.5-VL-3B · MindCube-Tiny | accuracy: 37.81 -> 32.00 | Hurt |
| add free-form reasoning at inference (ablation) | frozen Qwen2.5-VL-3B · MindCube-Tiny | accuracy: 37.81 -> 40.48 | Improved |
| generate a cognitive map before reasoning (ablation) | frozen Qwen2.5-VL-3B · MindCube-Tiny | accuracy: 40.48 -> 41.33 (plain map); Rotation 32.00 -> 25.00 | Mixed |
| SFT on raw QA pairs | Qwen2.5-VL-3B · MindCube-Tiny | accuracy: 37.81 -> 52.67 | Improved |
| SFT with augmented-map output only, instead of raw QA (ablation) | Qwen2.5-VL-3B SFT · MindCube-Tiny | accuracy: 52.67 -> 52.48 | No clear effect |
| SFT with free-form reasoning added (ablation) | Qwen2.5-VL-3B SFT · MindCube-Tiny | accuracy: 52.67 -> 55.43 | Improved |
| SFT to generate a plain cognitive map then reason (ablation) | Qwen2.5-VL-3B SFT · MindCube-Tiny | accuracy: 52.67 -> 57.81 | Improved |
| train RL from scratch with sparse reward instead of from SFT checkpoint (ablation) | RL-FFR · MindCube-Tiny | accuracy: 59.14 -> 49.52 | Hurt |
| initialize RL from the best SFT checkpoint (ablation) | RL-Plain-CGMap-FFR-Out · MindCube-Tiny | accuracy: 50.86 -> 61.33 | Improved |
| fix object order in cognitive map supervision instead of randomizing (ablation) | Plain-CGMap-FFR-Out SFT · MindCube-Tiny | accuracy: 57.81 -> 60.76 | Improved |
| use map-then-reason rather than reasoning only in RL from scratch (ablation) | RL from scratch · MindCube-Tiny | accuracy: 49.52 -> 52.48 (Aug), 50.86 (Plain) | Mixed |
Surprising findings
- Existing VLMs perform near random on limited-view spatial mental modeling questions.
Experimental setup
Benchmarks: MindCube
Metrics: accuracy
Closest papers by meaning
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.92)
- SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.91)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMs (ICLR 2026 · similarity 0.91)
- Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators (NeurIPS 2026 · similarity 0.91)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
