Research map / Vision-language and multimodal / Robot action and navigation
Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HSAN, a VLN-CE framework combining a VLM-built hierarchical semantic scene graph, an optimal-transport goal selector, and a GCN-based graph-aware PPO low-level controller, reporting SOTA on R2R-CE and RxR-CE.
Problem
Vision-language navigation in continuous environments (VLN-CE) struggles on long-horizon tasks due to limited scene understanding, inefficient planning, and weak decision-making under unseen environments.
Approach
Builds a multi-level (object, region, zone) semantic scene graph with VLM descriptions, selects long-term goals by solving an optimal-transport problem over ghost nodes balancing instruction relevance and geodesic distance, and executes subgoals with a GCN-augmented PPO policy.
Key results
- On R2R-CE val-unseen, HSAN reports SR 0.64 and SPL 0.59 versus LLaVA-Nav at 0.58 and 0.53.
- On RxR-CE val-unseen, HSAN reports SR 0.59 and SPL 0.54 versus LLaVA-Nav at 0.53 and 0.48.
- Ablations show SR drops of 5-8 points when removing the hierarchical graph, optimal transport, graph-aware control, or VLM descriptions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HSAN instead of LLaVA-Nav | VLN-CE navigation · R2R-CE validation unseen | SR: 0.58 -> 0.64 | Helped |
| Use HSAN instead of LLaVA-Nav | VLN-CE navigation · RxR-CE validation unseen | SR: 0.53 -> 0.59 | Helped |
| Use HSAN instead of LLaVA-Nav | VLN-CE navigation · RxR-CE multilingual subset | SR: 0.51 -> 0.57 | Helped |
| Use HSAN instead of LLaVA-Nav | VLN-CE navigation · R2R-CE high-clutter subset | SR: 0.54 -> 0.61 | Helped |
| Add hierarchical regions and zones to the object graph (ablation) | HSAN · R2R-CE validation unseen | SR: 0.57 -> 0.64 | Helped |
| Use optimal transport rather than a heuristic planner (ablation) | HSAN · R2R-CE validation unseen | SR: 0.59 -> 0.64 | Helped |
| Use graph-aware RL control rather than vanilla RL on raw visual features (ablation) | HSAN · R2R-CE validation unseen | SR: 0.56 -> 0.64 | Helped |
| Use VLM-generated descriptions rather than object category labels (ablation) | HSAN · R2R-CE validation unseen | SR: 0.58 -> 0.64 | Helped |
Surprising findings
- The reported main-table numbers for HSAN (SR 0.64 on R2R-CE) differ from the RxR-CE multilingual subset and high-clutter tables' HSAN figures for the same method, and the text reports inconsistent values for baselines across tables.
Limitations
- Computational overhead from real-time scene graph construction.
- Inference latency and reliance on VLM descriptions noted as future optimization targets.
Experimental setup
Datasets: R2R-CE RxR-CE Matterport3D
Benchmarks: R2R-CE RxR-CE
Metrics: SR SPL NE OSR
Baselines: CMM Waypoint Models Neural Topological SLAM Semantic MapNet GraphNav LLaVA-Nav
Builds on: Proximal Policy Optimization Sinkhorn algorithm Grounded-SAM SigLIP
Closest papers by meaning
- Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation (CVPR 2026 · similarity 0.94)
- LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation (NeurIPS 2026 · similarity 0.91)
- Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.89)
- Enhancing Multi-Robot Semantic Navigation Through Multimodal Chain-of-Thought Score Collaboration (AAAI 2025 · similarity 0.89)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.89)
- BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.