Research map / Vision-language and multimodal / Multimodal LLMs

D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes D3D-VLP, a single 3D vision-language-planning model that autoregressively generates a dynamic 3D chain-of-thought (planning, grounding, navigation, answer) with CoT memory feedback, trained with masked autoregressive loss on a 10M-sample fragmented hybrid dataset.

Problem

Embodied agents either use end-to-end models that lack interpretability and explicit 3D grounding, or modular systems whose planning, grounding, and navigation components are disjoint, lack dynamic replanning, and suffer domain gaps.

Approach

A unified autoregressive 3D-VLM generates planning text, grounded 3D tokens, waypoint navigation actions, and answers in one CoT sequence, fed back via a CoT memory for replanning; Synergistic Learning from Fragmented Supervision uses masked cross-entropy so partially annotated samples implicitly supervise other CoT components.

Key results

• Achieves 61.3% SR and 56.1% SPL on R2R-CE, beating StreamVLN (56.9% SR) and InternVLA-N1 (58.2% SR).
• On SG3D, reaches 33.7% s-SR and 13.8% t-SR navigation, and 9.3% t-ACC grounding versus 4.2% for Dynam3D-VisTA.
• Ablation: removing CoT Memory drops R2R-CE SR from 61.3% to 56.5% and SG3D t-ACC from 9.3% to 4.1%.

Surprising findings

• Training without planning annotations (types 4-6) gives 60.8% SR on R2R-CE, close to the full-data 61.3%.

Limitations

• SLFS relies on implicit gradient signals and autoregressive generation shows limited exploration; RL is suggested as future work.
• SG3D evaluation supplies ground-truth planning text, and the full planning pipeline is not reproduced.
• Training is limited by compute, so not all available data was used (sampled ~1:1:1 across data types).

Experimental setup

Datasets: SG3D Grounded 3D-LLM VLN-Trans SceneVerse MM-Scan PQ3D HM3D MP3D HSSD ProcTHOR-10K ProcTHOR-Objaverse R2R-CE

Benchmarks: R2R-CE REVERIE-CE NavRAG-CE HM3D-OVON SG3D

Metrics: NE SR OSR SPL s-SR t-SR s-ACC t-ACC

Baselines: StreamVLN NavFoM InternVLA-N1 Dynam3D MTU3D NaVILA Uni-NaVid OK-Robot DynaMem

Builds on: Dynam3D NVILA-Lite DAgger chain-of-thought

Closest papers by meaning

• Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation (NeurIPS 2025 · similarity 0.92)
• 3D-VLA: A 3D Vision-Language-Action Generative World Model (ICML 2024 · similarity 0.91)
• An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.90)
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models (CVPR 2026 · similarity 0.90)
• Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation (AAAI 2026 · similarity 0.90)
• MindJourney: Test-Time Scaling with World Models for Spatial Reasoning (NeurIPS 2025 · similarity 0.89)