Research map / Vision-language and multimodal / Multimodal LLMs

SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SpaCE-10, a benchmark of over 5k QA pairs on 811 indoor scenes that evaluates MLLMs on 10 atomic spatial capabilities combined into 8 compositional QA types, finding that even the best model (InternVL3.5-241B) trails humans (91.2%) by a wide margin.

Problem

Existing spatial intelligence benchmarks for MLLMs do not make atomic spatial capabilities explicit or systematically combine them into compositional tasks, and they are limited in scene count, modalities, and question types.

Approach

A hierarchical annotation pipeline (structured 2D/3D captions, GPT-4o basic QA generation, human filtering, then template-based capability integration) builds compositional QA from 10 atomic capabilities, with 2D image and 3D point cloud settings and single/multi-answer formats.

Key results

• Human experts score 91.2% overall on SpaCE-10, versus 53.1-53.4% for the best close-source model (GPT-5) and 55.0% for the best open-source model (InternVL3.5-241B-A28B).
• LEO-7B scores 11.1% overall versus 34.5% for GPT4Scene-7B, suggesting 2D MLLMs outperform the evaluated 3D MLLMs on this benchmark.
• Compositional QA types show large drops: EP accuracy falls from 67.8% to 26.8% when C7 and C9 are integrated, and FR falls from 83.0% to 42.8%.

Surprising findings

• Multiple-answer accuracy is far below single-answer accuracy; smaller InternVL2.5 models score below 5% on multiple-choice, below random selection.
• Counting (C5) is the weakest atomic capability across nearly all models, even for the largest InternVL3.5-241B (37.5%).

Limitations

• Human-expert hour counts and pipeline reliance on GPT-4o for QA generation may introduce generator bias.
• Evaluation uses only 8 frames as input and removes random answer assignment in LMMs-Eval, so results may underestimate some models relative to chance.
• Capability-level analysis relies on unweighted averages across mapped questions, and some human-vs-model claims are inconsistent across tables (e.g., GPT-5 scores 53.1 vs 53.4).

Experimental setup

Datasets: ScanNet ScanNet++ 3RScan ARKitScenes

Benchmarks: SpaCE-10

Metrics: accuracy capability score

Baselines: LEO GPT4Scene GPT-5 GPT-4o Claude-3.7-Sonnet Gemini-2.0-Flash-Thinking InternVL3.5-241B-A28B

Builds on: MMBench LMMs-Eval Sparkle

Closest papers by meaning

• SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.91)
• Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (CVPR 2025 · similarity 0.91)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.90)
• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.90)
• MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.89)
• See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model (NeurIPS 2025 · similarity 0.89)