Research map / Vision-language and multimodal / Multimodal LLMs

Grounded 3D-Aware Spatial Vision-Language Modeling

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Presents GR3D, a spatial VLM (NVILA-8B-Lite based) that adds explicit 2D grounding, implicit streaming region insertion during chain-of-thought, and region-prompted monocular 3D box prediction with intrinsic normalization and dense depth-derived point supervision.

Problem

Spatial VLMs lack implicit grounding of entity mentions during generation and struggle with monocular 3D grounding from a single image, where depth, scale, and intrinsics are entangled.

Approach

Insert region tokens into the text stream as the model generates entity mentions (streaming region insertion), and treat each grounded 2D region as a query for 3D box prediction, with intrinsic-aware image normalization and dense point supervision from depth maps.

Key results

• On Omni3D, GR3D-8B achieves 25.40 AP3D versus 24.92 for DetAny3D and roughly 17-18 mAP for Qwen3-VL-8B on several domains (Sun-RGBD mAP 31.64 vs 17.77).
• On MM-GCoT, GR3D-8B reaches 78.3 answer accuracy, 74.2 grounding accuracy, and 67.7 consistency on average, above LLaVA-GCoT-7B (74.5/63.3/58.1).
• On VSI-Bench, GR3D-8B averages 67.6 versus 62.9 for SR-3D-8B.

Surprising findings

• Qwen3-VL-8B's 3D detection is reported to be highly sensitive to input resolution because pixel dimensions implicitly encode focal length.
• Grounding improves general spatial VQA even without explicit localization outputs at inference.

Limitations

• Inference is slower than vision specialists due to LLM backbone, two-stage pipeline, and autoregressive 3D box generation.
• 3D training data (Omni3D, EmbodiedScan) covers limited environments, cameras, and categories.

Experimental setup

Datasets: RefSpatial OpenImages CA-1M Omni3D EmbodiedScan DepthLM Omni3D-CoT MM-GCoT VSI-Bench ScanRefer ScanQA

Benchmarks: Omni3D MM-GCoT BLINK CVBench ERQA SAT RWQA EMB ChartQA MME POPE AI2D

Metrics: AP15 mAP AP3D answer accuracy grounding accuracy consistency Acc@0.5 accuracy

Baselines: Cube R-CNN OVMono3D DetAny3D ImVoxelNet SMOKE Qwen3-VL-8B Qwen3-VL-4B VST SR-3D-8B LLaVA-GCoT-7B RoboRefer-8B

Builds on: SR-3D NVILA DetAny3D Florence-2 RefSpatial

Closest papers by meaning

• From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.93)
• G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.92)
• G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.92)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.92)
• Pursuing Minimal Sufficiency in Spatial Reasoning (ICLR 2026 · similarity 0.91)
• CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding (ICLR 2024 · similarity 0.91)