Research map / Vision-language and multimodal / Visual question answering
SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
SpatialPIN prompts and interacts with multiple 2D/3D foundation models to build an explicit 3D scene representation, enabling zero-shot, training-free spatial VQA and robotics planning with off-the-shelf VLMs.
Problem
VLMs have limited fine-grained spatial and 3D-aware reasoning; existing spatial VQA-trained VLMs (e.g., SpatialVLM) overfit to spatial VQA distributions and generalize poorly to higher-level 3D tasks like scene dynamics and trajectory planning.
Approach
A modular, training-free pipeline where a VLM progressively interacts with language-guided segmentation, inpainting, metric depth, perspective fields, and single-view 3D reconstruction to build a partial 3D scene, then receives summarized 3D object poses, sizes, and axes as text and visual prompts; perspective canonicalization and raycast-based placement are added for coarse-to-fine 3D grounding.
Key results
- GPT-4V qualitative IaOR-VQA accuracy rises from 70.7% to 86.3% with SpatialPIN; GPT-4o from 69.0% to 87.3%, versus SpatialVLM's 76.7%.
- On IrSD-VQA qualitative, GPT-4o improves from 64% to 82% with SpatialPIN, versus 54% for SpatialVLM.
- Pick-and-stack picked-and-stacked success rises to 36% with GPT-4o + ours, compared with 16% for direct 3D info and 10% for SpatialVLM + RRT*.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SpatialPIN's 3D priors and prompting to GPT-4o | GPT-4o · IaOR-VQA qualitative | Accuracy: 69.0 -> 87.3 | Helped |
| add SpatialPIN's 3D priors and prompting to GPT-4o | GPT-4o · IaOR-VQA quantitative | In range [75, 133] (%): 8.5 -> 70.5 | Helped |
| add SpatialPIN's 3D priors and prompting to GPT-4o | GPT-4o · IaAD-VQA qualitative | Accuracy (%): 68 -> 84 | Helped |
| add SpatialPIN's 3D priors and prompting to GPT-4o | GPT-4o · IrSD-VQA qualitative | Accuracy (%): 64 -> 82 | Helped |
| add SpatialPIN's 3D priors and prompting to GPT-4o | GPT-4o · IaAD-VQA quantitative | In range [75, 133] (%): 2 -> 42 | Helped |
| add SpatialPIN's 3D priors and prompting to GPT-4o | GPT-4o · IrSD-VQA quantitative | In range [75, 133] (%): 4 -> 54 | Helped |
| use SpatialPIN with GPT-4o reasoning rather than direct 3D information only | Pick-and-stack pipeline · Pick and stack | Picked & stacked (%): 16 -> 36 | Helped |
| add object descriptions and prompting for 2D object understanding (ablation) | SpatialPIN · IaOR-VQA qualitative | Accuracy: 68.3 -> 87.3 | Helped |
| add coarse 3D scene-size estimation and perspective canonicalization (ablation) | SpatialPIN · IaOR-VQA qualitative | Accuracy: 76.0 -> 87.3 | Helped |
| add fine-grained 3D scene understanding (ablation) | SpatialPIN · IaOR-VQA qualitative | Accuracy: 63.3 -> 87.3 | Helped |
| add coarse and fine-grained 3D understanding (ablation) | SpatialPIN · IaOR-VQA quantitative | Accuracy within [75, 133] (%): 58.0 -> 70.5 | Helped |
| replace SpatialPIN's pipeline with ShAPO (ablation) | SpatialPIN · IaOR-VQA qualitative | Accuracy: 87.3 -> 36.7 | Hurt |
Surprising findings
- Video-LLaVA-7B reported a 36.3% false positive rate on deliberately misaligned video-task pairs, so raw accuracy was adjusted by subtracting it.
- Humans described the generated task executions more accurately (OpenCLIP 0.823) than Video-LLaVA-7B did (0.636).
Limitations
- Inference is slow (~20 seconds per image) due to 3D object reconstruction and closed-source VLM API calls.
- Quantitative IaOR-VQA performance is suboptimal, with VLMs confusing camera and world coordinate frames.
- Evaluation datasets are small and hand-crafted (e.g., 20 image pairs for IrSD-VQA), and the pick-and-stack study uses only 50 simulated scenes.
Experimental setup
Datasets: NOCS RT-1 BridgeData V2 YCBInEOAT YCB
Benchmarks: IaOR-VQA IaAD-VQA IrSD-VQA ManiSkill pick and stack single-image task planning
Metrics: accuracy percentage within ratio range of ground truth pick/stack success rate user rating OpenCLIP similarity Self-BLEU embedding similarity
Baselines: SpatialVLM GPT-4V LLaVA-1.5 InstructBLIP ShAPO SAM-6D
Builds on: SpatialVLM Socratic Models One-2-3-45++ Language Segment-Anything RRT*
Closest papers by meaning
- Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.94)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (CVPR 2024 · similarity 0.92)
- SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning (NeurIPS 2025 · similarity 0.91)
- SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.