Research map / Vision-language and multimodal / Multimodal LLMs
SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SpatialRGPT, a region-aware VLM with a depth plugin trained on an automatically built 3D scene-graph dataset (OSD), plus a SpatialRGPT-Bench for 3D spatial VQA.
Problem
VLMs struggle with spatial reasoning (direction, distance, size) and cannot be prompted to reason about specific user-specified regions; they also lack 3D cues such as depth.
Approach
Automatically lift 2D OpenImages into 3D scene graphs (open-vocab detection, metric depth, camera calibration) to generate region-level spatial QAs, and add a plugin depth branch with its own connector (initialized from the RGB connector) to a region-aware VLM so depth is optional at inference.
Key results
- On SpatialRGPT-Bench, SpatialRGPT-VILA-1.5-8B reaches 92.69% average qualitative accuracy versus 58.14% for GPT-4V.
- On BLINK Relative Depth, SpatialRGPT-VILA-1.5-8B achieves 87.9% accuracy, above GPT-4V-Turbo at 66.9%.
- SpatialRGPT-VILA-1.5-3B matches or slightly exceeds VILA-1.5-3B on several general benchmarks (e.g., VQAv2 81.1 vs 80.4, MM-Vet 38.2 vs 35.4) while lowering MME (1424 vs 1442).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add relative-depth inputs to the RGB model (ablation) | SpatialRGPT-7B · SpatialRGPT-Bench qualitative average | success rate: 89.80 -> 91.78 | Helped |
| add metric-scale width and height training data (ablation) | SpatialRGPT-7B · SpatialRGPT-Bench size-related questions | success rate: 75.8 -> 80.1; 90.8 -> 91.9; 82.8 -> 87.5 | Helped |
| add metric-scale width and height training data (ablation) | SpatialRGPT-7B · SpatialRGPT-Bench horizontal and vertical distance | success rate: 69.3 -> 65.6; 54.8 -> 51.9 | Hurt |
| replace axis-aligned boxes with PCA-based oriented boxes (ablation) | 3D scene graph pipeline · Omni3D test set | width MSE: 8.27 -> 17.09 | Hurt |
| replace axis-aligned boxes with PCA-based oriented boxes (ablation) | 3D scene graph pipeline · Omni3D test set | height MSE: 2.35 -> 4.83 | Hurt |
| use box inputs instead of mask inputs (ablation) | SpatialRGPT-7B · SpatialRGPT-Bench qualitative average | success rate: 91.78 -> 91.47 | No clear effect |
| train with spatial VQA data in SpatialRGPT rather than the base VILA model | SpatialRGPT-VILA-1.5-3B · VQA-v2 | score: 80.4 -> 81.1 | Helped |
Surprising findings
- Blind LLMs with language referral achieve commendable success rates on some quantitative width/height questions, suggesting world knowledge carries much of the signal.
- Axis-aligned bounding boxes outperform PCA-based oriented boxes from single-view point clouds in width/height MSE (8.27 vs 17.09).
Limitations
- Axis-aligned bounding boxes can misrepresent object orientation and label accuracy.
- Single-view point clouds are noisy, limiting precise 3D measurements.
- Evaluation relies on GPT-4 as judge, and the benchmark is small (657 qualitative and 749 quantitative QAs).
Experimental setup
Datasets: OpenImages CC3M MMC4 COYO Omni3D nuScenes KITTI SUNRGBD ARKitScenes Hypersim COCO LVIS
Benchmarks: SpatialRGPT-Bench COCO-2017 region classification BLINK relative depth VQAv2 GQA SQA-I VQA-T POPE MME MMBench SEED MMMU
Metrics: success rate absolute relative error accuracy mAP
Baselines: GPT-4 GPT-4V LLaVA-v1.6-34B KOSMOS-2 RegionVILA-7B GPT-4V+SoM VILA-1.5-3B RegionGPT-7B GPT-4V-Turbo GPT-4o
Builds on: RegionGPT VILA SpatialVLM Metric3Dv2 WildCamera Llama 3
Closest papers by meaning
- SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.94)
- SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.94)
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.94)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.93)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.93)
- InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.