Research map / Vision-language and multimodal / Multimodal LLMs

Abstract 3D Perception for Spatial Intelligence in Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SandboxVLM, a training-free pipeline that uses video-diffusion multi-view priors, VLM-guided segmentation, proxy elevation, and multi-view voting to build abstract 3D bounding boxes that are rendered and fed to VLMs for zero-shot spatial reasoning.

Problem

VLMs struggle with 3D spatial cognition and physical understanding because they are trained on 2D images and text; existing 3D-LLMs need dense 3D supervision and cannot be applied to proprietary models.

Approach

Represent a scene as a compact set of abstract 3D oriented bounding boxes lifted from multi-view video-diffusion priors via proxy elevation and cross-view voting, then render these abstractions (step-back and top-down views) as extra visual context for a frozen VLM.

Key results

• SandboxVLM reaches 84.1% on SAT-Real with GPT-5-mini, versus 75.4% for the vanilla backbone and 78.7% for MindJourney.
• Average spatial accuracy of 81.4% across spatial benchmarks, above GPT-5-mini (78.5%) and Gemini-2.5-Pro (80.3%) in the main table.
• On PhysBench, SandboxVLM scores 58.3%, 3.4 points above MindJourney (54.9%).

Surprising findings

• Directly rendering point clouds from reconstructed scenes lowers accuracy below the vanilla VLM baseline (73.7% vs 75.4%).
• 3D box coordinates given as text outperform rendered proxy points as visual input, suggesting current VLMs are language-centric.

Limitations

• Static scene assumption: the 3D sandbox does not model dynamic objects or correlations across views, hurting object-movement questions.
• No explicit physical properties (mass, friction, material) are modeled.
• Modular pipeline errors from VLM pointing, segmentation, and depth/camera estimation propagate to the final answer.

Experimental setup

Datasets: SAT BLINK EmbSpatial-Bench PhysBench

Benchmarks: SAT-Real SAT-Synth BLINK-Spatial BLINK-Depth EmbSpatial-Bench PhysBench

Metrics: accuracy

Baselines: GPT-4o GPT-5-mini Claude-Sonnet-4 Gemini-2.5-Pro Qwen2.5-VL-72B MindJourney RoboBrain2.0-32B VeBrain-8B

Builds on: MindJourney SEVA VGGT SAM DBSCAN

Closest papers by meaning

• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.92)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.92)
• HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models (CVPR 2026 · similarity 0.92)
• G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.91)
• Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.91)
• Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes (ICLR 2026 · similarity 0.91)