Research map / Vision-language and multimodal / Robot action and navigation
World-aware Planning Narratives Enhance Large Vision-Language Model Planner
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WAP, a world-aware narrative enhancement framework that augments ALFRED instructions and step-wise reasoning across four cognitive dimensions and trains LVLM planners with curriculum learning using only visual observations, reaching 62.7 average success on EB-ALFRED with Qwen2.5-VL-7B.
Problem
LVLM embodied planners trained with environment-agnostic imitation learning struggle with context-sensitive instructions, unfamiliar environments, and long-horizon multi-step tasks, often relying on privileged feedback rather than visual reasoning.
Approach
Augment expert ALFRED instruction-trajectory pairs with visual, spatial, functional, and syntactic instruction variants and generated stepwise reasoning, filter them by semantic verification, and train with a three-stage curriculum so the planner uses only raw egocentric images and instructions in closed loop.
Key results
- Qwen2.5-VL-7B with curriculum learning reaches 62.7 average success on EB-ALFRED, up from 4.7 for the base model.
- InternVL3-8B with curriculum learning reaches 61.0 average success, up from 10.7 for the base model.
- Long-horizon success for Qwen2.5-VL-7B reaches 70, compared with 2 for the base model and 52 for... (Claude-3.5-Sonnet closed-loop reports 74 on long tasks).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add WAP with full augmentation and curriculum learning | Qwen2.5-VL-7B-Ins · EB-ALFRED | average success rate: 4.7 -> 62.7 | Helped |
| add WAP with full augmentation and curriculum learning | InternVL3-8B · EB-ALFRED | average success rate: 10.7 -> 61.0 | Helped |
| add basic reasoning (ablation) | Qwen2.5-VL-7B-Ins · EB-ALFRED | average success rate: 4.7 -> 47.0 | Helped |
| restrict augmentation to visual and spatial dimensions (ablation) | Qwen2.5-VL-7B-Ins + Basic Reasoning · EB-ALFRED | average success rate: 47.0 -> 46.7 | Hurt |
| use partial rather than complete reasoning replacement (ablation) | Qwen2.5-VL-7B-Ins · EB-ALFRED | average success rate: 54.0 -> 58.0 | Hurt |
| replace partial reasoning with complete reasoning (ablation) | Qwen2.5-VL-7B-Ins · EB-ALFRED | STD: 9.3 -> 6.8 | Helped |
| add curriculum learning to full augmentation (ablation) | Qwen2.5-VL-7B-Ins · EB-ALFRED | average success rate: 58.0 -> 62.7 | Helped |
| add curriculum learning to full augmentation (ablation) | Qwen2.5-VL-7B-Ins · EB-ALFRED Long | success rate: +12 over Complete Reasoning | Helped |
| use self-directed rather than explicit curriculum enhancement | Qwen2.5-VL-7B-Ins · EB-ALFRED | average success rate: 62.7 -> 56.7 | Hurt |
| use self-directed rather than explicit curriculum enhancement | Qwen2.5-VL-7B-Ins · EB-ALFRED Common | success rate: 48 vs. 62 | Hurt |
| use self-directed rather than explicit curriculum enhancement | Qwen2.5-VL-7B-Ins · EB-ALFRED Spatial | success rate: 52 vs. 52 | No clear effect |
Surprising findings
- Self-directed enhancement without explicit augmentation rules reaches 56.7 average, close to explicit enhancement.
- Visual/spatial-only augmentation does not improve average performance (46.7) and increases variance (STD 17.1).
Limitations
- Operates at symbolic action level without continuous control.
- Generalization to industrial or outdoor settings not verified; evaluated only on household ALFRED.
- Limited capacity for mid-execution error correction.
Experimental setup
Datasets: ALFRED EB-ALFRED
Benchmarks: EB-ALFRED
Metrics: success rate standard deviation across task categories
Baselines: GPT-4o Claude-3.5-Sonnet Gemini-1.5-Pro InternVL2.5-78B-MPO Qwen2.5-VL-72B-Ins InternVL3-8B
Builds on: ALFRED EmbodiedBench
Closest papers by meaning
- Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.91)
- Evaluating Vision-Language Models as Evaluators in Path Planning (CVPR 2025 · similarity 0.90)
- APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts (NeurIPS 2026 · similarity 0.90)
- Agent Planning with World Knowledge Model (NeurIPS 2024 · similarity 0.89)
- World–Value–Action Model: Implicit Planning for Vision–Language–Action Systems (NeurIPS 2026 · similarity 0.88)
- Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.