Research map / Vision-language and multimodal / Multimodal LLMs
Does Spatial Cognition Emerge in Frontier Models?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SPACE, a benchmark of 10 large-scale and small-scale spatial cognition tasks adapted from cognitive science, with text and image presentations, showing frontier LLMs/VLMs perform near chance on many tasks.
Problem
Whether spatial cognition emerges in disembodied frontier models trained on text, images, and video; existing benchmarks cover only limited spatial reasoning and rarely mirror classic animal and human cognition experiments.
Approach
A benchmark that operationalizes classic cognitive-science spatial tests (large-scale navigation/mapping and small-scale visuospatial tasks) with parallel text (character-array) and image presentations so both LLMs and VLMs can be evaluated, plus control tests checking that models understand the inputs.
Key results
- On ego-image large-scale tasks, GPT-4o averages 23.0% versus 15.0% chance and human performance of 82.8-96.6% on multiple-choice tasks.
- On small-scale multimodal tasks, Claude 3.5 Sonnet averages 43.8% and GPT-4o 40.1%, while text-only averages reach 64.5% and 65.2%.
- Models exceed 95% on selective attention (SAtt) matching or surpassing human performance (95%), yet remain near chance on perspective taking and MPFB with multimodal input.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use text-only presentations rather than multimodal presentations | GPT-4o and Claude 3.5 Sonnet · Small-scale spatial cognition overall | average accuracy: 40.1 -> 65.2; 43.8 -> 64.5 | Helped |
| Use DM image rather than ego image observations | GPT-4o · Large-scale spatial cognition | average accuracy/SPL: 23.0 -> 28.8 | Helped |
| Use DM text rather than ego image observations | GPT-4o · Large-scale spatial cognition | average accuracy/SPL: 23.0 -> 32.6 | Helped |
| Use DM text rather than DM image observations | GPT-4o · Large-scale spatial cognition | average accuracy/SPL: 28.8 -> 32.6 | Helped |
| Use text-only rather than multimodal presentation | GPT-4o · Selective attention (SAtt) | accuracy: 70.2 ± 1.8 -> 98.8 ± 0.4 | Helped |
| Use text-only rather than multimodal presentation | GPT-4o · Mental rotation (MRT) | accuracy: 33.3 ± 1.9 -> 41.9 ± 6.2 | Helped |
| Use text-only rather than multimodal presentation | GPT-4o · Perspective taking (PTT) | accuracy: 26.5 ± 3.6 -> 55.5 ± 3.9 | Helped |
| Use text-only rather than multimodal presentation | GPT-4o · Minnesota Paper Form Board (MPFB) | accuracy: 27.0 ± 2.2 -> 50.5 ± 9.6 | Helped |
| Use text-only rather than multimodal presentation | GPT-4o · Judgement of Line Orientation (JLO) | accuracy: 26.5 ± 5.9 -> 66.5 ± 4.8 | Helped |
| Use text-only rather than multimodal presentation | GPT-4o · Maze completion (MCT) | success rate: 10.4 ± 1.0 -> 21.5 ± 3.8 | Helped |
| Use text-only rather than multimodal presentation | GPT-4o · Corsi block-tapping (CBTT) | accuracy: 68.0 ± 2.0 -> 82.5 ± 1.7 | Helped |
| Use text-only rather than multimodal presentation | GPT-4o · Spatial addition (SAdd) | accuracy: 40.5 ± 7.1 -> 93.5 ± 3.6 | Helped |
Surprising findings
- Frontier models match or exceed human accuracy on the selective attention task while performing near chance on mental rotation and maze completion.
- Models are near chance on large-scale tasks even with allocentric map views, yet understand the same inputs well in control tests.
Limitations
- Human baselines were collected only for multiple-choice tasks with 29 participants.
- Text-only versions of MRT, MPFB, and JLO are simplified and not directly comparable to visual versions.
- Some multimodal open models ran out of memory on MCT and CSWM, with accuracy set to 0.
Experimental setup
Datasets: ImageNet
Benchmarks: SPACE
Metrics: accuracy SPL success rate
Baselines: Human Chance Claude 3.5 Sonnet GPT-4o GPT-4v Mistral 123B Llama 3 70B
Builds on: PlanBench CogEval EWOK MMStar ARC
Closest papers by meaning
- Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests (NeurIPS 2026 · similarity 0.90)
- SpatialTree: How Spatial Intelligence Branches Out in MLLMs (CVPR 2026 · similarity 0.88)
- Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations (ICLR 2026 · similarity 0.88)
- From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs (CVPR 2026 · similarity 0.88)
- Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models (ICML 2026 · similarity 0.87)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.