Research map / Vision-language and multimodal / Multimodal LLMs
Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces STARE, a ~4K-task benchmark testing multimodal LLMs on spatial reasoning that benefits from multi-step visual simulation, finding models near chance on cube-net folding and tangram puzzles while humans are near-perfect.
Problem
Existing multimodal benchmarks mostly test static recognition or linguistically re-phrasable reasoning, leaving non-verbal, multi-step visual simulation under-evaluated in MLLMs.
Approach
A programmatically generated benchmark spanning 2D/3D geometric transformations, cube net folding, tangram puzzles, temporal frame and perspective reasoning, with controlled settings with/without explicit intermediate visual simulations, compared against human performance and response times.
Key results
- o1 achieves the best overall accuracy of 57.2%, while humans reach 96.5% overall.
- Cube net folding and tangram puzzles stay near chance (about 50%) for most models, while 2D transformations reach up to 87.7% (o1 with visual simulation).
- Visual simulation helps most models on 2D/3D transformations but degrades some cases, e.g., GPT-4o and o1 on tangram and Gemini-2.0 Flash on cube nets.
Surprising findings
- Providing explicit verbal steps lowers GPT-4o tangram accuracy from 62.4% to 34.7%, suggesting shortcut behavior such as summing piece areas.
- Gemini-2.0 Flash Thinking underperforms its non-reasoning counterpart Gemini-2.0 Flash on several tasks.
Limitations
- Synthetic, simplified images that do not capture real-world clutter and occlusion.
- Focuses only on rigid transformations; no flexible or articulated objects.
- Multiple-choice and yes/no scoring hides intermediate reasoning steps.
Experimental setup
Datasets: Objectron HM3D
Benchmarks: STARE
Metrics: accuracy F1 macro-average
Baselines: Random Claude-3.5 Sonnet Gemini-2.0 Flash Gemini-2.0 Flash Thinking o1 LLaVA-OneVision-72B InternVL2.5-78B Qwen2.5-VL-72B
Builds on: Objectron HM3D Habitat Blender
Closest papers by meaning
- Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests (NeurIPS 2026 · similarity 0.92)
- Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing (NeurIPS 2025 · similarity 0.92)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.91)
- From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs (CVPR 2026 · similarity 0.91)
- SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.91)
- 3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models (ICML 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.