Research map / Vision-language and multimodal / Multimodal LLMs

MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMSI-Bench, a 1,000-question human-curated multi-image spatial reasoning VQA benchmark, and evaluates 37 MLLMs, finding a large gap to humans (97%) with the best model around 40%.

Problem

Existing spatial benchmarks probe mainly single-image relations and template-generated questions, so they do not measure multi-image spatial reasoning that real-world embodied deployments require.

Approach

A fully human-authored multiple-choice VQA benchmark where each question requires integrating two (or more, for multi-step) real-world images, with distractors, annotated stepwise reasoning, and an automated error-type analysis that uses those reasoning annotations.

Key results

• Best open-source model Qwen2.5-VL-72B reaches 30.7% average accuracy; best proprietary GPT-5 reaches 41.9% vs. 97.2% human.
• Blind GPT-4o scores 22.7%, close to random guessing (25%), indicating visual reasoning is required.
• Zero-shot CoT helps only GPT-4o marginally and degrades other tested models; visual prompting with PATS correspondences gives mixed results.

Surprising findings

• Scaling model size within a series yields limited gains (Qwen2.5-VL-72B only ~3% above 32B; InternVL3-78B only 1.5% above InternVL3-1B).
• Spatially fine-tuned models show only marginal gains over their base models on MMSI-Bench.

Limitations

• Manual annotation limits scalability; benchmark has 1,000 questions.
• Automated error analysis is only about 78% accurate even with reference reasoning, and error labels rely on an LLM judge.
• Multi-image setup is a proxy; video-based spatial intelligence is not evaluated.

Experimental setup

Datasets: Matterport3D ScanNet DTU nuScenes Waymo AgiBot-World DAVIS 2017 Ego4D

Benchmarks: MMSI-Bench

Metrics: accuracy

Baselines: Random Guessing Human Level Blind GPT-4o Spatial-MLLM InternSpatial RoboBrain2.0

Builds on: VLMEvalKit PATS

Closest papers by meaning

• SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.93)
• Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.93)
• Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces (ICML 2026 · similarity 0.92)
• Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.92)
• Scaling Spatial Intelligence with Multimodal Foundation Models (CVPR 2026 · similarity 0.92)
• InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.92)