Research map / Vision-language and multimodal / Multimodal LLMs
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates 2B spatial VQA pairs from 10M internet images using off-the-shelf vision experts (CLIP filtering, FlexCap captions, ZoeDepth metric depth) and trains a PaLM 2-E-based VLM on them to gain quantitative and qualitative 3D spatial reasoning.
Problem
VLMs struggle with 3D spatial reasoning such as qualitative spatial relations and metric distances/sizes, likely because internet-scale image-caption training data contains little spatial information.
Approach
Automatically lift internet-scale 2D images into metric 3D point clouds using off-the-shelf expert models (open-vocabulary captioning, segmentation, metric depth, horizontal-plane canonicalization), then template-generate millions of direct spatial VQA pairs and co-train a VLM on them alongside PaLM-E data.
Key results
- SpatialVLM reaches 75.2% accuracy on binary spatial predicate questions versus 71.3% for LLaVA-1.5 and 68.0% for GPT-4V.
- SpatialVLM outputs valid numeric answers 99.0% of the time and falls within the [50,200]% range for 37.2% of quantitative questions, versus 33.9% for PaLM 2-E.
- Co-training improves VQA v2 by 2.4 points (76.6% to 79.0%) while OKVQA drops slightly (61.4% to 61.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add synthetic spatial VQA data (co-training) to PaLM 2-E training mix | PaLM 2-E based VLM · Qualitative spatial VQA benchmark | Accuracy: 50.4% to 75.2% | Helped |
| Add synthetic spatial VQA data (co-training) to PaLM 2-E training mix | PaLM 2-E based VLM · Quantitative spatial VQA benchmark | In range [50,200]%: 33.9% to 37.2% | Helped |
| Add synthetic spatial VQA data (co-training) to PaLM 2-E training mix | PaLM 2-E based VLM · VQA v2 test-dev | Accuracy: 76.6% to 79.0% (+2.4) | Helped |
| Add synthetic spatial VQA data (co-training) to PaLM 2-E training mix | PaLM 2-E based VLM · OKVQA | Accuracy: 61.4% to 61.0% (-0.4) | No clear effect |
| Unfreeze the ViT encoder during finetuning on spatial data (ablation) | SpatialVLM (PaLM 2-S) · Manipulation quantitative VQA | In range [50,200]%: 34.9% to 37.2% (+2.3) | Helped |
| Unfreeze the ViT encoder during finetuning on spatial data (ablation) | SpatialVLM (PaLM 2-S) · Manipulation quantitative VQA | In range [90,110]%: 5.6% to 8.4% (+2.8) | Helped |
| Add Gaussian label noise (std 0 to 0.3) to quantitative answers (ablation) | SpatialVLM manipulation quantitative training · Manipulation quantitative VQA | In range [50,200]%: 59.0% (std 0) vs 61.1% (std 0.3) | No clear effect |
| Use LLM (GPT-4) orchestrating SpatialVLM for chain-of-thought spatial reasoning | SpatialVLM · Complex spatial reasoning examples | qualitative demonstration | No clear effect |
Surprising findings
- Adding Gaussian noise to quantitative training labels (std up to 0.3 m) leaves half-to-twice accuracy roughly unchanged, suggesting the model learns spatial common sense despite noisy labels.
- Unfreezing the contrastively pretrained ViT improves fine-grained distance estimation, indicating the frozen encoder is lossy for fine spatial information.
Limitations
- Accuracy is bounded by the biases and metric range of monocular expert models such as ZoeDepth; errors grow for larger distances.
- Quantitative labels are noisy and human benchmark annotations show limited agreement.
- Questions are built on a finite set of templates and only cover up to two objects per question.
Experimental setup
Datasets: WebLI VQAv2 PaLM-E training mix robotic manipulation dataset
Benchmarks: Spatial VQA human-annotated benchmark (qualitative and quantitative) OKVQA VQA v2
Metrics: accuracy output numbers rate percentage in half-to-twice range of ground truth MSE
Baselines: GPT-4V LLaVA-1.5 InstructBLIP PaLI PaLM-E PaLM 2-E
Builds on: PaLM-E PaLM 2 ZoeDepth FlexCap Socratic Models
Closest papers by meaning
- SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models (CVPR 2026 · similarity 0.94)
- SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.92)
- Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes (ICLR 2026 · similarity 0.92)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.92)
- SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors (NeurIPS 2024 · similarity 0.92)
- Enhancing Spatial Reasoning Through Visual and Textual Thinking (AAAI 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.