Research map / Vision-language and multimodal / Multimodal LLMs

From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OpenBench, an open-world pedestrian-view video benchmark with metric ground truth from stereo, LiDAR and IMU/GPS, showing MLLM spatial gains on indoor benchmarks do not transfer and that models rely on linguistic priors.

Problem

Existing spatial-intelligence benchmarks for MLLMs are mostly qualitative or limited to indoor scenes with mesh-based ground truth, and lack outdoor datasets with verifiable metric and kinematic ground truth.

Approach

Builds an outdoor video QA benchmark from multi-sensor pedestrian-perspective recordings (stereo cameras, LiDAR, IMU/GPS) with SLAM poses, depth maps, and detector/tracker-based 3D object trajectories, generating template-based questions across relational, metric, and kinematic tiers, plus blinding and synthetic abnormal-scene tests to probe linguistic priors.

Key results

• Human-model disparity is largest on relational tasks (relative direction: humans 83.3 vs 23-30 for top models).
• Vision-enabled gains over vision-disabled are marginal for most MLLMs (e.g., GPT +2.2, Qwen3-VL +5.3) versus +22.6 for humans.
• Newer InternVL3.5 models underperform InternVL2 on OpenBench absolute distance despite large gains on VSI-Bench (+24.1 for InternVL3.5-38B over InternVL2-40B on VSI-Bench).

Surprising findings

• Newer InternVL3.5 models consistently underperform their InternVL2 predecessors on absolute distance in open-world settings despite large indoor gains.
• Gemini-2.5-Pro size estimation drops from 54.7 to 28.3 on synthetic abnormal scenes, indicating reliance on class-level size priors.

Limitations

• Human-annotated and MLLM-curated QA pairs depend on an automated pipeline with calibration and SLAM errors; object full 3D dimensions are not estimated.
• Ground-truth 3D positions rely on LiDAR-densified depth and object centroids, with measured mean positional error of 0.68 m indoor and 0.79 m outdoor.

Experimental setup

Datasets: OpenBench (self-collected stereo/LiDAR/IMU pedestrian videos) VSI-Bench

Benchmarks: OpenBench VSI-Bench

Metrics: accuracy mean relative accuracy (MRA)

Baselines: VSI-Bench STI-Bench Gemini-2.5-Pro GPT-4o Qwen3-VL-32B InternVL3.5

Builds on: ORB-SLAM3 GroundingDINO SAM CoTracker3 VLMEvalKit

Closest papers by meaning

• Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes (ICLR 2026 · similarity 0.92)
• MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.92)
• InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.92)
• Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.92)
• SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.92)
• Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.92)