Research map / Vision-language and multimodal / Multimodal LLMs
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VSI-Bench, a video-based visual-spatial intelligence benchmark of ~5,000 QA pairs from indoor scan videos, evaluates 15 MLLMs against humans, and probes linguistic (CoT, self-consistency, ToT) and visual (cognitive map) reasoning.
Problem
Whether multimodal LLMs trained on large video data can perceive, remember, and recall 3D spatial layouts from egocentric video, which existing video benchmarks do not measure.
Approach
A video-based benchmark (VSI-Bench) built from 3D-annotated indoor scan datasets with template-generated QA over eight spatial tasks, a new mean relative accuracy metric for numerical answers, and probing via self-explanations, prompting-based reasoning methods, and LLM-generated cognitive maps.
Key results
- Gemini-1.5 Pro is the best model on VSI-Bench, trailing human evaluators (79% average accuracy) by about 33%.
- Zero-shot CoT, self-consistency, and Tree-of-Thoughts fail to improve average VSI-Bench performance; Zero-shot CoT and ToT drop average performance by about 4%.
- Generating cognitive maps raises Gemini-1.5 Pro relative distance accuracy from 46.0% to 56.0%, and the MLLM's cognitive map accuracy is 64% for adjacent objects but falls with distance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate and use a cognitive map | Gemini-1.5 Pro · VSI-Bench relative distance | accuracy: 46.0 -> 56.0; improves ... by 10% | Helped |
| Use a ground-truth cognitive map instead of the model-predicted map (ablation) | Gemini-1.5 Pro · VSI-Bench relative distance | accuracy: 56.0 -> 66.0 | Helped |
| Increase the MLLM cognitive map canvas from 10 × 10 to 20 × 20 (ablation) | Gemini-1.5 Pro · VSI-Bench relative distance | accuracy: 56.0 -> 54.0 | Hurt |
| Increase the ground-truth cognitive map canvas from 10 × 10 to 20 × 20 (ablation) | Gemini-1.5 Pro · VSI-Bench relative distance | accuracy: 66.0 -> 78.0 | Helped |
| Generate and use a cognitive map | LLaVA-Video-72B · VSI-Bench relative distance | accuracy: 36.0 -> 42.0; 8% performance gain | Helped |
| Generate and use a cognitive map | LLaVA-Video-7B · VSI-Bench relative distance | accuracy: 40.0 -> 32.0 | Hurt |
| Add zero-shot chain-of-thought prompting | Gemini-1.5 Pro · VSI-Bench average | performance: about 4% | Hurt |
| Add self-consistency with chain-of-thought | Gemini-1.5 Pro · VSI-Bench average | performance: 1.1% | Hurt |
| Add tree-of-thoughts prompting | Gemini-1.5 Pro · VSI-Bench average | performance: about 4% | Hurt |
| Add zero-shot chain-of-thought prompting | Gemini-1.5 Pro · VideoMME 500-question subset | performance: 77.2 -> 79.8; 1.6% | Helped |
| Use frame captions from LLaVA-Video-72B and GPT-4o to answer instead of standard GPT-4o | GPT-4o · VSI-Bench average | performance: 34.0 -> 29.3; 4.7% | Hurt |
| Present the video before the question rather than the question first (ablation) | Gemini-1.5 Pro · VSI-Bench (tiny) average | performance: 2.5% | Hurt |
Surprising findings
- Linguistic prompting techniques (CoT, self-consistency, ToT) harm spatial reasoning despite helping general video and visual tasks.
- Blind (no-video) models perform below chance level on several tasks, and most open-source models score below chance frequency baseline.
Limitations
- Benchmark is built from only 288 indoor scan videos, limiting environment diversity.
- Human error analysis on 163 samples and cognitive map evaluation are small-scale manual analyses.
- Results depend on frame sampling choices and default prompts per model.
Experimental setup
Datasets: ScanNet ScanNet++ ARKitScenes
Benchmarks: VSI-Bench VideoMME
Metrics: accuracy mean relative accuracy
Baselines: Gemini-1.5 Flash GPT-4o InternVL2 VILA-1.5 LongVILA LongVA LLaVA-OneVision LLaVA-Video Chance level (random) Chance level (frequency) Human
Builds on: Zero-shot CoT Self-Consistency Tree-of-Thoughts VideoMME LMMs-Eval
Closest papers by meaning
- SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence (ICLR 2026 · similarity 0.91)
- SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.91)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.91)
- Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World (CVPR 2026 · similarity 0.90)
- Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.90)
- From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.