Research map / Vision-language and multimodal / Multimodal LLMs

Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SSI-Bench, a human-curated 1,000-question ranking VQA benchmark for structure-centric spatial reasoning in constraint-governed real-world 3D structures, and evaluates 31 VLMs against humans.

Problem

Existing spatial benchmarks target unconstrained scenes where single-image 3D relations are underdetermined; there is no benchmark for spatial reasoning over real-world structures governed by geometric, topological, and physical constraints.

Approach

Use structural constraints (geometry, topology, physical feasibility) of real-world 3D structures to make component relations determinate, and pose multi-candidate permutation ranking questions over geometric and topological criteria, with a fully human-curated pipeline.

Key results

• Best closed-source model Gemini-3-Flash reaches 33.60% average Taskwise Accuracy, versus 91.60% for humans and 12.85% for random guessing.
• Best open-source model GLM-4.6V reaches 22.20% average Taskwise Accuracy.
• Thinking yields only modest gains: Gemini-3-Pro improves from 27.1% (low) to 29.5% (high); Qwen3-VL-30B-A3B from 20.6% (Instruct) to 22.5% (Thinking).

Surprising findings

• Chain-of-thought / extra thinking tokens give only marginal gains, and accuracy is non-monotonic in token usage.
• Scaling within model families yields inconsistent gains; larger variants do not always improve substantially.

Limitations

• Benchmark is limited to 1,000 questions and ranking-format multiple-choice questions.
• Human-centered curation is costly (400+ researcher-hours), limiting scaling; semi-automatic expansion only proposed.

Experimental setup

Datasets: Unsplash Pexels Pixabay original photography

Benchmarks: SSI-Bench

Metrics: Taskwise Accuracy Pairwise Accuracy

Baselines: Random Guessing Human Performance

Builds on: Enact

Closest papers by meaning

• MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.92)
• SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.92)
• Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.92)
• 3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models (ICML 2026 · similarity 0.92)
• SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.92)
• SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models (CVPR 2026 · similarity 0.91)