Research map / Large language models / Retrieval and knowledge
Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Following
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ExRAP, a retrieval-augmented LLM planner that uses a temporal embodied knowledge graph memory, entropy-based temporal consistency refinement, and exploration-integrated planning with information-gain estimates to handle continual conditional instructions in non-stationary VirtualHome, ALFRED, and CARLA environments.
Problem
Continual instruction following for embodied agents, where multiple conditional tasks depend on a changing environment, requiring efficient exploration and up-to-date environmental memory without repeated full re-observation.
Approach
Combine a temporal embodied knowledge graph memory with LLM-based query evaluation (with entropy-based temporal consistency to counter memory decay) and a planner that sums an in-context exploitation value with an information-gain-based exploration value to select skills.
Key results
- In VirtualHome under medium non-stationarity, ExRAP reaches 55.14% SR and 11.33 PS, versus 39.89% SR and 15.93 PS for LLM-Planner.
- In ALFRED under low non-stationarity, ExRAP reaches 69.90% SR and 5.94 PS, versus 58.44% SR and 7.28 PS for LLM-Planner.
- ExRAP with Gemma-2B achieves 52.75% SR, while LLM-Planner with Gemma-2B achieves 23.31% SR, showing robustness to smaller models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ExRAP instead of LLM-Planner | Continual instruction following · VirtualHome, medium non-stationarity | SR: 39.89% ± 4.52% -> 55.14% ± 6.59% | Helped |
| Use ExRAP instead of LLM-Planner | Continual instruction following · ALFRED, medium non-stationarity | SR: 51.80% ± 3.79% -> 64.00% ± 5.07% | Helped |
| Use ExRAP instead of LLM-Planner | Continual instruction following · CARLA, medium non-stationarity | SR: 44.00% ± 0.70% -> 62.25% ± 6.72% | Helped |
| Add temporal consistency refinement to query evaluation (ablation) | ExRAP · VirtualHome, medium non-stationarity, 5 instructions | SR: 43.92% ± 7.97% -> 55.14% ± 6.59% | Helped |
| Add temporal consistency refinement to query evaluation (ablation) | ExRAP · VirtualHome, medium non-stationarity, 5 instructions | PS: 15.47 ± 2.17 -> 11.33 ± 1.92 | Helped |
| Use information-based exploration instead of LLM-directed exploration (ablation) | ExRAP · VirtualHome, medium non-stationarity, 5 instructions | SR: 29.42% ± 25.63% -> 55.14% ± 6.59% | Helped |
| Use information-based exploration instead of LLM-directed exploration (ablation) | ExRAP · VirtualHome, medium non-stationarity, 5 instructions | PS: 10.78 ± 3.84 -> 11.33 ± 1.92 | Mixed |
| Integrate exploration with exploitation instead of exploitation-only planning (ablation) | ExRAP · VirtualHome, medium non-stationarity, 5 instructions | SR: 40.68% ± 9.77% -> 55.14% ± 6.59% | Helped |
| Integrate exploration with exploitation instead of exploitation-only planning (ablation) | ExRAP · VirtualHome, medium non-stationarity, 5 instructions | PS: 13.03 ± 3.01 -> 11.33 ± 1.92 | Mixed |
Experimental setup
Datasets: VirtualHome expert trajectories (100 across 10 settings) ALFRED-based expert trajectories (50) CARLA trajectories (50)
Benchmarks: VirtualHome ALFRED CARLA
Metrics: success rate (SR) pending step (PS)
Baselines: ZSP SayCan ProgPrompt LLM-Planner
Builds on: DREAM LLM-Planner SayCan ProgPrompt ZSP
Closest papers by meaning
- ThinkBot: Embodied Instruction Following with Thought Chain Reasoning (ICLR 2025 · similarity 0.87)
- Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation (AAAI 2025 · similarity 0.87)
- Embodied Task Planning via Graph-Informed Action Generation with Large Language Models (ICML 2026 · similarity 0.87)
- Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization (ICLR 2026 · similarity 0.86)
- Retrieval-Augmented Embodied Agents (CVPR 2024 · similarity 0.86)
- Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.