Research map / Vision-language and multimodal / Multimodal LLMs
Spatial Understanding from Videos: Structured Prompts Meet Simulation Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpatialMind, a structured CoT prompting strategy (scene and question decomposition), and ScanForgeQA, a 925K-QA synthetic dataset from simulated 3D scans, to improve vision-only 3D spatial reasoning in pre-trained VLMs.
Problem
Vision-only 3D spatial reasoning from scanning videos is hard due to spatial uncertainty (no explicit depth) and scarce, non-scalable real-scan training data for VLMs.
Approach
Combine a structured prompting scheme that decomposes scenes into local/global maps, grids, and textual descriptions and decomposes questions into type-specific reasoning steps, with a scalable pipeline that renders simulated indoor scenes (3D-FRONT, HoloDeck) into scan videos and auto-generates spatial QA for SFT.
Key results
- Qwen2.5-VL-7B average on VSI-Bench rises from 37.2 to 43.9 with SpatialMind+ScanForgeQA (+6.7%), versus 39.2 with prompting alone (+2.0%) and 43.3 with fine-tuning alone.
- Qwen2.5-VL-72B reaches 47.1 average on VSI-Bench with both methods (+7.9%), the largest gain for that model.
- On OpenEQA EM-EQA, Qwen2.5-VL-7B improves from 50.1 to 58.6 accuracy with both methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SpatialMind and fine-tune on ScanForgeQA | Qwen2.5-VL-7B · VSI-Bench | Avg accuracy: 37.2 -> 43.9 | Helped |
| Add SpatialMind and fine-tune on ScanForgeQA | Qwen2.5-VL-72B · VSI-Bench | Avg accuracy: 39.2 -> 47.1 | Helped |
| Add SpatialMind and fine-tune on ScanForgeQA | Qwen2.5-VL-7B · OpenEQA EM-EQA | Acc/Score: 50.1/3.1 -> 58.6/3.5 | Helped |
| Add SpatialMind and fine-tune on ScanForgeQA | Qwen2.5-VL-7B · ScanQA validation | BLEU-1: 32.5 -> 37.9 | Helped |
| Add SpatialMind and fine-tune on ScanForgeQA | Qwen2.5-VL-7B · SQA3D validation | EM-1: 17.2 -> 24.5 | Helped |
| Add SpatialMind prompting only (ablation) | Qwen2.5-VL-7B · VSI-Bench | Avg accuracy: 37.2 -> 39.2 | Helped |
| Fine-tune on ScanForgeQA only (ablation) | Qwen2.5-VL-7B · VSI-Bench | Avg accuracy: 37.2 -> 43.3 | Helped |
| Fine-tune on SQA3D instead of ScanForgeQA (ablation) | Qwen2.5-VL-7B · VSI-Bench | Room Size accuracy: 38.9 -> 38.8 | Hurt |
| Fine-tune on ScanQA instead of ScanForgeQA (ablation) | Qwen2.5-VL-7B · VSI-Bench | Room Size accuracy: 38.9 -> 38.5 | Hurt |
| Use question reasoning component only instead of full SpatialMind (ablation) | Qwen2.5-VL-72B · VSI-Bench | Avg accuracy: 44.0 -> 41.3 | Hurt |
| Use scene description component only instead of full SpatialMind (ablation) | Qwen2.5-VL-72B · VSI-Bench | Avg accuracy: 44.0 -> 42.7 | Hurt |
| Fine-tune on ScanForgeQA | Qwen2.5-VL-7B · Video-MME | performance | Hurt |
Surprising findings
- Text-based object position descriptions were more interpretable to VLMs than a 3D map or 2D grid representation.
- Fine-tuning on ScanForgeQA slightly lowered Video-MME performance, and fine-tuning on existing ScanQA or SQA3D reduced Room Size accuracy.
Limitations
- Relies on textual descriptions, which may not capture spatial semantics as well as visual input.
- Focuses on single-room scenes; multi-room, large-scale, and outdoor settings are underexplored.
- Synthetic data may limit transfer to real-world scans.
Experimental setup
Datasets: ScanForgeQA 3D-FRONT HoloDeck Objaverse SQA3D ScanQA ShareGPT4Video MVBench Video-MME
Benchmarks: VSI-Bench OpenEQA ScanQA SQA3D MVBench Video-MME
Metrics: accuracy MRA-style tolerance accuracy LLM-judge score BLEU-1 EM-1
Baselines: Gemini-1.5 Pro GPT-4o InternVL2-8B InternVL2-40B Qwen2.5-VL-7B Qwen2.5-VL-72B SQA3D fine-tuning ScanQA fine-tuning
Builds on: chain-of-thought prompting 3D-FRONT HoloDeck VSI-Bench
Closest papers by meaning
- SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors (NeurIPS 2024 · similarity 0.94)
- Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs (NeurIPS 2025 · similarity 0.93)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.93)
- SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning (NeurIPS 2025 · similarity 0.92)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.