Research map / Vision-language and multimodal / Multimodal LLMs

Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Ego3D-Bench, an 8.6K-QA benchmark for spatial reasoning of VLMs on ego-centric multi-view outdoor driving images, and Ego3D-VLM, a training-free plug-in that builds a textual cognitive map from REC and metric depth to improve VLM answers.

Problem

Current VLMs struggle with 3D spatial reasoning over ego-centric multi-view observations (e.g., from self-driving cars or robots), and no benchmark evaluates this setting.

Approach

Convert referred objects detected by a REC model and depth-estimated 3D positions across all views into a textual cognitive map in a front-camera-centered global coordinate frame, with relational scaling via common-sense object heights, and feed this map with the images to any VLM.

Key results

• Ego3D-VLM with GPT-4o raises average multiple-choice accuracy from 56.7% to 73.2% and cuts distance RMSE from 19.2 to 7.4 m.
• Ego3D-VLM with Qwen2.5-72B reaches 69.5% average accuracy versus 58.0% for the base model, and InternVL3-78B reaches 71.8% versus 59.9%.
• Human level averages 85.3% on Ego3D-Bench, and the best Ego3D-VLM configuration remains about 14 points below it.

Surprising findings

• Blind VLMs (text only) score only about 5% lower than vision-enabled VLMs on the benchmark.
• Ego3D-VLM with GPT-4o performs better than the GT-bbox visual prompt setting for the base model, suggesting 3D perception rather than 2D perception is the main bottleneck.

Limitations

• Performance depends on the reasoning ability of the underlying VLM; small models such as VILA1.5-8B show little gain.
• REC outputs include all prompt-mentioned objects, adding redundant or misleading information to the cognitive map.
• Metric depth in outdoor scenes is inaccurate; relational scaling with common-sense heights is approximate.

Experimental setup

Datasets: nuScenes Waymo Open Dataset Argoverse 1

Benchmarks: Ego3D-Bench All-Angle Bench VSI-Bench

Metrics: accuracy RMSE

Baselines: GPT-4o Gemini-1.5-Pro Qwen2.5-VL InternVL3 Ovis2 SpatialRGPT SpaceThinker-Qwen2.5-3B SpaceQwen2.5-3B All-Angle Bench VSI-Bench

Builds on: Grounding DINO Depth Anything V2 SpatialRGPT SpatialVLM

Closest papers by meaning

• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.92)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.92)
• InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.92)
• SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (CVPR 2024 · similarity 0.92)
• EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy (CVPR 2026 · similarity 0.91)
• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.91)