Research map / Vision-language and multimodal / Robot action and navigation
Affordances-Oriented Planning Using Foundation Models for Continuous Vision-Language Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AO-Planner, a zero-shot two-stage LLM framework for continuous VLN that uses SAM-based navigational affordances and visual affordances prompting to plan low-level paths, then a PathAgent to select them, and distills LLM waypoints into a learned predictor.
Problem
Existing LLM-based VLN agents handle only high-level planning over predefined navigation graphs, while continuous VLN (VLN-CE) requires low-level motion planning that typically needs simulator-trained waypoint predictors.
Approach
Segment visible ground with Grounded SAM as navigational affordances, overlay numbered points and candidate paths on the image so a multimodal LLM can select waypoints and plan obstacle-avoiding paths in RGB space, then lift pixel paths to 3D using depth and camera intrinsics; a second LLM agent (PathAgent) chooses among the marked paths.
Key results
- Zero-shot AO-Planner reaches 25.5% SR and 16.6% SPL on R2R-CE val-unseen, and 22.4% SR and 15.1% SPL on RxR-CE, outperforming the prior best zero-shot A^2Nav (22.6 SR / 11.1 SPL on R2R-CE).
- Distilled waypoint predictor combined with ETPNav's VLN agent reaches 47% SR on R2R-CE, competing with supervised methods (AO-Planner supervised variant row in Table 1).
- Removing navigational affordances (raw scattered points) drops SR from 27% to 15% on a 100-episode R2R-CE subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AO-Planner instead of A²Nav | Zero-shot VLN-CE · R2R-CE validation unseen | SR: 22.6 -> 25.5 | Helped |
| Use AO-Planner instead of A²Nav | Zero-shot VLN-CE · RxR-CE validation unseen | SPL: 6.3 -> 15.1 | Helped |
| Add instructions to VAP (ablation) | AO-Planner low-level prompting · R2R-CE validation unseen, 100-case ablation subset | SR: 24 -> 27; 3% improvement | Helped |
| Add waypoint definitions, including obstacle avoidance and crucial-region guidance (ablation) | AO-Planner low-level prompting · R2R-CE validation unseen, 100-case ablation subset | SR: 23 -> 27; 4% reduction in SR without definitions | Helped |
| Use segmented navigational affordances (ablation) | AO-Planner low-level prompting · R2R-CE validation unseen, 100-case ablation subset | SR: 15% to 27% | Helped |
| Use PathAgent instead of Baseline Agent (ablation) | Zero-Shot Predictor · R2R-CE validation unseen, 100-case ablation subset | SR: 20 -> 27 | Helped |
| Replace zero-shot predictor with supervised predictor (ablation) | PathAgent · R2R-CE validation unseen, 100-case ablation subset | SPL: 16.9 vs. 17.6 | Helped |
| Use Gemini-1.5-Pro for low-level planning and GPT-4o for high-level decisions (ablation) | AO-Planner LLM agents · R2R-CE validation unseen, 100-case ablation subset | SR: 11 -> 27 | Helped |
| Distill AO-Planner waypoint predictions into a learning-based predictor | ETPNav VLN agent · R2R-CE validation unseen | SR: 47 vs. ETPNav 57 | Mixed |
Surprising findings
- Gemini-1.5-Pro performed better than GPT-4o for low-level waypoint planning, while GPT-4o was better for high-level decisions.
- Pseudo-labeling with Gemini for 40K images costs about 84 USD, versus roughly 20K USD if the zero-shot LLM were queried during VLN agent fine-tuning.
Limitations
- Zero-shot performance still lags supervised methods substantially (e.g., ETPNav 57% SR on R2R-CE).
- About 70% of failures come from high-level decision errors and 30% from low-level planning errors such as misdetected ground and collisions.
- Relies on expensive closed-source API calls (GPT-4o, Gemini-1.5-Pro) and a 100-episode subset was used for ablations to save cost.
Experimental setup
Datasets: R2R-CE RxR-CE
Benchmarks: R2R-CE RxR-CE
Metrics: NE OSR SR SPL nDTW
Baselines: A^2Nav ZSON Cow CLIP-Nav NaVid ETPNav DC-VLN MapGPT
Builds on: Set-of-Mark prompting Grounded SAM MapGPT ETPNav
Closest papers by meaning
- Evaluating Vision-Language Models as Evaluators in Path Planning (CVPR 2025 · similarity 0.90)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.89)
- An Interactive Navigation Method with Effect-oriented Affordance (CVPR 2024 · similarity 0.89)
- D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation (CVPR 2026 · similarity 0.88)
- Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.88)
- World–Value–Action Model: Implicit Planning for Vision–Language–Action Systems (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.