Research map / 3D vision / 3D generation
From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SPAR-7M, a 7M-QA 2D spatial dataset generated from 3D ground truth of indoor scans, and the SPAR-Bench benchmark, and shows that fine-tuning InternVL2.5-8B on it (mixed with general 2D data) improves spatial perception and reasoning without 3D inputs.
Problem
Vision-language models struggle with spatial perception and reasoning about 3D scenes, and existing spatial datasets and benchmarks focus on semantics or lack precise 3D ground truth and multi-view coverage.
Approach
Generate 2D spatial QA data (single-view, multi-view, and spatial imagination tasks) directly from 3D scene ground truth (meshes, boxes, camera poses) so VLMs learn 3D-aware spatial understanding from images alone, with no 3D inputs at inference.
Key results
- InternVL2.5-8B+SPAR-mix reaches 41.1 on VSI-Bench (vs 32.4 base, 24.5 +EMOVA-2M) and 89.08 on CV-Bench 3D (vs 78.50 base).
- On SPAR-Bench, InternVL2.5-8B+SPAR-mix averages 63.25 versus 36.39 for GPT-4o and 41.74 for the best full-benchmark baseline Doubao-1.5-vision-pro; human level is 67.27.
- BEV coordinate prediction APE mean drops from 2.22 (base) to 1.05 (SPAR-mix) on ScanNet++ validation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on SPAR-mix instead of EMOVA-2M | InternVL2.5-8B · VSI-Bench | score: 24.5 -> 41.1 | Helped |
| Pretrain on SPAR-mix instead of EMOVA-2M | InternVL2.5-8B · CV-Bench 3D | score: 64.83 -> 89.08 | Helped |
| Pretrain on SPAR-mix instead of EMOVA-2M | InternVL2.5-8B · CV-Bench 2D | score: 66.27 -> 72.25 | Helped |
| Pretrain on SPAR-mix | InternVL2.5-8B · SPAR-Bench full | average score: 36.28 -> 63.25 | Helped |
| Pretrain on SPAR-mix | InternVL2.5-8B · ScanNet++ val BEV coordinates | APE Mean: 2.22 -> 1.05 | Helped |
| Reduce SPAR-7M proportion from 60% to 40% (ablation) | InternVL2.5-4B · CV-Bench 2D | score: 73.00 → 74.13 | Helped |
| Reduce SPAR-7M proportion from 60% to 40% (ablation) | InternVL2.5-4B · CV-Bench 3D | score: 91.10 → 89.09 | Hurt |
| Reduce SPAR-7M proportion from 60% to 40% (ablation) | InternVL2.5-4B · SPAR-Bench medium | score: 44.04 → 42.21 | Hurt |
| Reduce SPAR-7M proportion from 60% to 40% (ablation) | InternVL2.5-4B · SPAR-Bench high | score: 51.86 → 48.19 | Hurt |
| Reduce image resolution from 448×448 to 224×224 (ablation) | SPAR-mix · ScanQA | CiDEr: 90.7 to 87.6 | Hurt |
Surprising findings
- Fine-tuned model surpasses human performance on depth prediction in SPAR-Bench (MRA-based metrics).
- Base InternVL2.5-8B + general EMOVA-2M data alone drops VSI-Bench from 32.4 to 24.5, so general-data-only training hurts spatial skills.
Limitations
- Static, image-centric QA does not capture continuous or interactive embodied reasoning.
- Monocular setting has scale ambiguity for metric tasks, and the pipeline needs 3D ground truth (depth, intrinsics, poses) that may be unavailable.
Experimental setup
Datasets: SPAR-7M EMOVA ScanNet ScanNet++ Structured3D ScanQA SQA3D ScanRefer CV-Bench VSI-Bench Hypersim SUNRGBD
Benchmarks: SPAR-Bench CV-Bench VSI-Bench BLINK 3DSRBench Seed-Image MME MMBench RealWorldQA TextVQA ScanQA SQA3D
Metrics: accuracy MRA APE EM BLEU-4 CiDEr Acc@0.25 Acc@0.5
Baselines: GPT-4o GPT-4.1 Qwen2.5-VL-72B InternVL2.5-8B Video-3D LLM LLaVA-3D Scene-LLM LEO Chat-3D v2 3D-LLM
Builds on: InternVL2.5 EMOVA ScanNet ScanNet++ Structured3D
Closest papers by meaning
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.93)
- Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.93)
- Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models (CVPR 2026 · similarity 0.93)
- SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.93)
- Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.93)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.