Research map / Large language models / Agents, code and math
Scene Map-based Prompt Tuning for Navigation Instruction Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAPINSTRUCTOR, an LLM-based navigation instruction generator that adds 3D voxel scene encoding, topological map prompt tuning, and semantic-entropy landmark filtering, reporting gains on R2R, REVERIE, and RxR.
Problem
Navigation instruction generation (NIG) needs accurate 3D spatial understanding and global map context, which prior 2D/BEV and text-only map methods lack, leading to missing landmarks and hallucinated landmarks.
Approach
Encodes egocentric panoramas into 3D voxel features, integrates a GCN-encoded topological map of the trajectory as soft prompt tokens into a parameter-efficiently tuned LLM, and uses semantic entropy over multiple sampled landmark lists to filter hallucinated landmarks before instruction completion.
Key results
- On R2R val unseen, CIDEr reaches 0.490 vs 0.449 baseline and the paper reports 4.0% CIDEr gain over competitors on unseen split.
- On REVERIE val unseen, CIDEr 0.562 vs 0.489 for BEVInstructor (3.8% gain); SPL with HAMT 24.12% approaches human annotation (30.20%).
- User study (30 students, 100 instructions) gives MAPInstructor an average score of 4.11 vs 3.91 for BEVInstructor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MAPINSTRUCTOR instead of the strongest baseline, BEVINSTRUCTOR | NIG models · R2R val seen | CIDEr: 0.549 -> 0.576 | Helped |
| use MAPINSTRUCTOR instead of the strongest baseline, C-INSTRUCTOR | NIG models · R2R val unseen | CIDEr: 0.450 -> 0.490 | Helped |
| use MAPINSTRUCTOR instead of the strongest baseline, C-INSTRUCTOR | NIG models · REVERIE val seen | CIDEr: 0.753 -> 0.771 | Helped |
| use MAPINSTRUCTOR instead of the strongest baseline, C-INSTRUCTOR | NIG models · REVERIE val unseen | CIDEr: 0.524 -> 0.562 | Helped |
| use MAPINSTRUCTOR instead of the strongest baseline, C-INSTRUCTOR | NIG models · RxR val seen | CIDEr: 0.056 -> 0.075 | Helped |
| use MAPINSTRUCTOR instead of the strongest baseline, C-INSTRUCTOR | NIG models · RxR val unseen | CIDEr: 0.053 -> 0.057 | Helped |
| add scene representation encoding (ablation) | MAPINSTRUCTOR vanilla baseline · R2R val unseen | CIDEr: 0.449 -> 0.465 | Helped |
| add map prompt tuning (ablation) | MAPINSTRUCTOR vanilla baseline · R2R val unseen | CIDEr: 0.449 -> 0.476 | Helped |
| add landmark uncertainty assessment (ablation) | MAPINSTRUCTOR vanilla baseline · R2R val unseen | CIDEr: 0.449 -> 0.460 | Helped |
| combine scene representation encoding, map prompt tuning, and landmark uncertainty assessment (ablation) | MAPINSTRUCTOR · R2R val unseen | CIDEr: 0.449 -> 0.490 | Helped |
| replace BEV features with a 3D scene representation (ablation) | MAPINSTRUCTOR · R2R val unseen | CIDEr: 0.479 -> 0.490 | Helped |
| vary the mapping architecture among GraphSAGE, GCN, and GAT (ablation) | MAPINSTRUCTOR · R2R val unseen | CIDEr: 0.486; 0.490; 0.485 | No clear effect |
Surprising findings
- Increasing landmark-prediction rounds beyond M=3 gives marginal gains, with M=5 slightly lower than M=3 on SPICE and Rouge.
Limitations
- Relies on GPT-4 to extract ground-truth landmark labels for training.
- Gains on some metrics (SPICE, Meteor, Rouge on REVERIE) are not best, and scene map is limited to static indoor environments.
Experimental setup
Datasets: R2R REVERIE RxR PREVALENT
Benchmarks: R2R REVERIE RxR
Metrics: BLEU CIDEr METEOR ROUGE SPICE SR SPL
Baselines: BT-speaker EDrop-speaker CCC-speaker Lana Lana+ C-INSTRUCTOR BEVInstructor
Builds on: BEVInstructor Lana semantic entropy deformable attention GCN
Closest papers by meaning
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.90)
- STRIDER: Navigation via Instruction-Aligned Structural Decision Space Optimization (NeurIPS 2025 · similarity 0.89)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.89)
- Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator (NeurIPS 2026 · similarity 0.89)
- Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation (AAAI 2026 · similarity 0.88)
- LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.