Research map / Vision-language and multimodal / Robot action and navigation
Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Dynam3D builds a dynamic, layered patch-instance-zone 3D representation from posed RGB-D with CLIP features and feeds it, with a LLaVA-Phi-3-mini 3D-VLM, to predict navigation actions for monocular VLN.
Problem
Video-VLM-based monocular vision-and-language navigation struggles with 3D geometry/semantics, large-scale exploration, long-term memory, and dynamic changing scenes.
Approach
Project CLIP patch features into 3D with depth and poses, maintain them with frustum-culling updates, group them into instances (via a learned 2D-3D merging discriminator) and cubic zones, align them to CLIP/language space via distillation and contrastive losses, and feed the multi-level tokens to a 3D-VLM that predicts atomic navigation actions.
Key results
- On R2R-CE test (monocular), Dynam3D reaches 51.4% SR and 44.8 SPL, versus 46.3% SR for g3D-LF and 47.0% SR for Uni-NaVid on val.
- On REVERIE-CE val, Dynam3D reports 40.1% SR, over 13 points above retrained NaVid (26.6%).
- Real-world static tests with Hello Robot Stretch 3 reach 70% SR after pre-exploration versus 35% for NaVid.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use hierarchical 3D patch-instance-zone tokens instead of the strongest reported baseline | Dynam3D · R2R-CE Val | SR: 47.2 -> 52.9 | Helped |
| use hierarchical 3D patch-instance-zone tokens instead of the strongest reported baseline | Dynam3D · R2R-CE Test | SR: 46.3 -> 51.4 | Helped |
| use hierarchical 3D patch-instance-zone tokens instead of the strongest reported baseline | Dynam3D · REVERIE-CE Val | SR: 34.4 -> 40.1 | Helped |
| use hierarchical 3D patch-instance-zone tokens instead of the strongest reported baseline | Dynam3D · NavRAG-CE Val | SR: 21.4 -> 24.7 | Helped |
| add 3D instance and zone representations (ablation) | Dynam3D · R2R-CE Val Unseen | SR: 45.7 -> 52.9 | Helped |
| add 3D instance and zone representations (ablation) | Dynam3D · REVERIE-CE Val | SR: 25.7 -> 40.1 | Helped |
| add 3D zone representations (ablation) | Dynam3D with instance tokens · REVERIE-CE Val | SR: 39.3 -> 40.1 | Helped |
| add subspace alignment supervision (ablation) | Dynam3D with instance and zone tokens · R2R-CE Val Unseen | SR: 50.7 -> 52.9 | Helped |
| add subspace alignment supervision (ablation) | Dynam3D with instance and zone tokens · REVERIE-CE Val | SR: 38.4 -> 40.1 | Helped |
| add pre-exploration | Dynam3D · R2R-CE Val | SR: 52.9 -> 57.1 | Helped |
| add lifelong memory | Dynam3D · R2R-CE Val | SR: 52.9 -> 55.6 | Helped |
| add pre-exploration and lifelong memory | Dynam3D · REVERIE-CE Val | SR: 40.1 -> 49.5 | Helped |
Limitations
- Does not explicitly output target instance coordinates, limiting use in mobile manipulation.
- Lacks question answering, dialogue, and task update capabilities.
- Real-world evaluation uses only 20 test cases per setting.
Experimental setup
Datasets: ScanNet HM3D Matterport3D 3RScan ARKitScenes Structured3D SceneVerse ScaleVLN NavRAG R2R-CE REVERIE-CE
Benchmarks: R2R-CE REVERIE-CE NavRAG-CE
Metrics: NE OSR SR SPL
Baselines: NaVid Uni-NaVid g3D-LF VLN-3DFF InstructNav AO-Planner CM2 WS-MGMap
Builds on: g3D-LF LLaVA FastSAM Feature Splatting ETPNav DynaMem
Closest papers by meaning
- D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation (CVPR 2026 · similarity 0.92)
- GA-VLN: Geometry-Aware BEV Representation for Efficient Vision-Language Navigation (CVPR 2026 · similarity 0.90)
- Volumetric Environment Representation for Vision-Language Navigation (CVPR 2024 · similarity 0.90)
- Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation (AAAI 2026 · similarity 0.89)
- g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks (CVPR 2025 · similarity 0.89)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.