Research map / Vision-language and multimodal / Multimodal LLMs

Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DSR Suite (an automated pipeline producing DSR-Train and human-refined DSR-Bench from in-the-wild videos) and GSM, a dual Q-Former module that selects question-relevant pi^3 geometry tokens for Qwen2.5-VL-7B, improving dynamic spatial reasoning without hurting general video understanding.

Problem

VLMs are weak at dynamic spatial reasoning (reasoning about how object geometry and relations evolve in 3D over time), largely due to scarce scalable 4D-aware training data, and naively injecting 3D reconstruction features degrades general video understanding.

Approach

An automated pipeline that uses vision foundation models (pi^3, Grounded SAM2, Orient Anything) to extract camera poses, point clouds, object masks, orientations and 3D trajectories from in-the-wild videos to generate procedural multiple-choice DSR QA; plus GSM, which uses two stacked Q-Formers to condense question semantics and extract only question-relevant geometry into a fixed set of tokens.

Key results

• The proposed model (Qwen2.5-VL-7B + GSM trained on DSR-Train) reaches 58.9 average on DSR-Bench versus 23.5 for the base Qwen2.5-VL-7B and 38.4 for VG-LLM.
• GSM achieves 50.2 average across DSR-Bench, VLM4D, STI-Bench and Video-MME, versus 40.0 for the baseline, while keeping Video-MME at 59.9 (baseline 60.2).
• Naive Addition of pi^3 tokens reaches 57.7 on DSR-Bench but drops Video-MME to 48.6.

Surprising findings

• Naively adding 3D reconstruction tokens improves dynamic spatial reasoning but noticeably degrades general video understanding (Video-MME 60.2 to 48.6).
• Spatial reasoning models trained only on static scenes (VLM-3R, VG-LLM) outperform proprietary models such as GPT-4o and Gemini-2.5-Flash on DSR-Bench.

Limitations

• Geometry is relative-scale, so questions are restricted to qualitative trend-based answers rather than metric values.
• Non-template questions and benchmark labels rely on DeepSeek-R1 and Gemini-2.5-Pro, and 3D requirement and granularity checks are LLM-judged.

Experimental setup

Datasets: Koala-36M DSR-Train VLM-3R static data Cambrian-S static data MineRL

Benchmarks: DSR-Bench VLM4D STI-Bench Video-MME VSI-Bench MineDojo

Metrics: accuracy

Baselines: GPT-4o GPT-5 Gemini-2.5-Pro Qwen2.5-VL-7B Qwen3-VL-8B-Instruct VLM-3R VG-LLM LLaVA-Video-7B

Builds on: Koala-36M Grounded SAM2 Orient Anything pi^3 Qwen2.5-VL

Closest papers by meaning

• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.93)
• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.92)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.92)
• Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.91)
• Multi-view Relational Distillation for Spatial Reasoning with Vision-Language Models (NeurIPS 2026 · similarity 0.91)
• SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning (NeurIPS 2025 · similarity 0.91)