Research map / Vision-language and multimodal / Multimodal LLMs
SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the MSMU dataset (700K metric-annotated spatial QA pairs from ScanNet/ScanNet++) and a depth positional encoding that adds sinusoidal depth embeddings to CLIP image features of a LLaVA-1.5-7B model, yielding SD-VLM.
Problem
VLMs struggle with quantitative 3D spatial reasoning (absolute sizes, distances) from 2D images, partly because of scarce precisely annotated quantitative spatial data and the lack of 3D information in image inputs.
Approach
Generate large-scale metric spatial QA from 3D scene data with physical scales (rather than model-estimated labels), and inject depth into the VLM by adding sinusoidal depth positional embeddings (from pooled depth per patch) to the vision features, avoiding extra tokens or APIs.
Key results
- SD-VLM reaches 56.31% average success rate on MSMU-Bench, versus 30.82% for Qwen2.5-VL-72B-level baselines and 33.63% for Intern-VL3-78B (abstract reports +26.91 over GPT-4o and +25.56 over Intern-VL3-78B).
- With MSMU-CoT training, SD-VLM average on MSMU-Bench rises to 59.19%.
- SD-VLM achieves 56.2% on Q-Spatial++ and 33.3% quantitative / 65.5% qualitative on SpatialRGPT-Bench, the best among listed baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MSMU-CoT training pairs (ablation) | SD-VLM · MSMU-Bench | average success rate: 56.31 to 59.19 | Helped |
| add MSMU-CoT training pairs (ablation) | SD-VLM · MSMU-Bench scale comparison | success rate: 55.66% to 69.81% | Helped |
| use DPE rather than no depth input (ablation) | LLaVA-1.5-7B · MSMU-Bench | success rate: 46.73 to 56.31 | Helped |
| encode depth maps as images (ablation) | LLaVA-1.5-7B · MSMU-Bench | success rate: 46.73 to 22.64 | Hurt |
| provide depth as a textual prompt (ablation) | LLaVA-1.5-7B · MSMU-Bench | success rate: 46.73 to 48.78 | Helped |
| concatenate depth embeddings as tokens (ablation) | LLaVA-1.5-7B · MSMU-Bench | success rate: 46.73 to 35.72 | Hurt |
| add DPE during general instruction tuning (ablation) | LLaVA-1.5-7B · Q-Spatial++ | score: 11.2 to 23.0 | Helped |
| replace DepthAnything with UniDepth (ablation) | SD-VLM · MSMU-Bench | success rate: 56.3% vs. 56.2% | No clear effect |
| replace DepthAnything with UniDepth (ablation) | SD-VLM · average across spatial benchmarks | average performance: 48.6% to 47.6% | Hurt |
| add Gaussian noise to depth maps (ablation) | SD-VLM · MSMU-Bench | success rate: 56.3% under no noise to 51.4% | Hurt |
| use estimated rather than ground-truth depth at both training and inference (ablation) | SD-VLM · MSMU-Bench | success rate: 57.71 to 55.35 | Hurt |
| remove depth input (ablation) | SD-VLM · MSMU-Bench | success rate: 56.31 to 46.73 | Hurt |
Surprising findings
- Depth-as-image integration drops MSMU-Bench success from 46.73% to 22.64%, worse than no depth.
- Training with estimated depth (Depth-Anything-V2) yields 55.35%, close to the 56.31% obtained with ground-truth depth at inference.
Limitations
- MSMU focuses on indoor scenes, limiting applicability to outdoor or dynamic settings.
- Base model is small (LLaVA-1.5-7B); larger base models not explored.
- Evaluation relies on GPT-4-Turbo for answer extraction and scoring, adding judge dependence.
Experimental setup
Datasets: ScanNet ScanNet++ LLaVA-1.5-mix665k
Benchmarks: MSMU-Bench Q-Spatial++ SpatialRGPT-Bench Whatsup GQA TextVQA VQA-v2 Vizwiz
Metrics: success rate (threshold delta) accuracy
Baselines: GPT-4o Gemini-2 Qwen2.5-VL-72B Intern-VL3-78B SpatialBot SpatialRGPT LLaVA-1.5-7B
Builds on: LLaVA-1.5 SpatialRGPT Depth-Anything-V2 sinusoidal positional encoding
Closest papers by meaning
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.93)
- Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.93)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models (CVPR 2026 · similarity 0.92)
- G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning (CVPR 2026 · similarity 0.92)
- SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (CVPR 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.