Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Multi-Modal Grounded Planning and Efficient Replanning for Learning Embodied Agents with a Few Examples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FLARE, a few-shot embodied planner for ALFRED that retrieves in-context examples using both instruction and visual similarity and replans by swapping unavailable objects with semantically similar observed ones, without calling the LLM again.
Problem
Task planning for embodied household agents from natural language instructions with very few annotated examples; LLM planners often ignore the environment state at command time and produce ungrounded plans when target objects are missing or referred to with ambiguous wording.
Approach
Multi-modal planner retrieves in-context examples using a weighted sum of instruction similarity (BERT) and egocentric view similarity (CLIP), and an environment adaptive replanner replaces unreachable target objects with the most semantically similar observed object via language-embedding cosine similarity, avoiding extra LLM calls.
Key results
- In the few-shot (0.5%) setting with GPT-4, FLARE reaches 40.88 SR and 51.72 GC on test unseen (goal + sequential instructions), versus 23.37 GC for LLM-Planner with GPT-3.5-era baselines and up to +24.46 absolute gain claimed.
- With GPT-3.5, FLARE w/o EAR planner accuracy rises to 55.66 unseen versus 31.67 for LLM-Planner static planning.
- Ablation on GPT-3.5 shows removing MMP drops unseen SR from 31.79 to 30.35 and removing EAR drops it to 30.28, with both removed giving 28.58.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FLARE-GPT-4 instead of LLM-Planner | Few-shot ALFRED agent · ALFRED Test Unseen, Goal instructions + Sequential instructions | SR: 16.42 -> 40.88 | Helped |
| use FLARE-GPT-4 instead of LLM-Planner | Few-shot ALFRED agent · ALFRED Test Unseen, Goal instruction only | SR: 13.41 -> 32.57 | Helped |
| add MMP (multimodal example retrieval) (ablation) | FLARE-GPT-3.5 · ALFRED Test Unseen | SR: 30.35 -> 31.79 | Helped |
| add MMP (multimodal example retrieval) (ablation) | FLARE-GPT-3.5 · ALFRED Test Unseen | GC: 42.40 -> 43.94 | Helped |
| add EAR (visual adaptive replanning) (ablation) | FLARE-GPT-3.5 · ALFRED Test Unseen | SR: 30.28 -> 31.79; 1.51 %p | Helped |
| add EAR (visual adaptive replanning) (ablation) | FLARE-GPT-3.5 · ALFRED Test Unseen | GC: 42.48 -> 43.94 | Helped |
Surprising findings
- LLaMA2-13B with FLARE reaches only 0.006 planner accuracy for LLM-Planner static, showing open models can fail badly without retrieval, though FLARE-LLaMA2 still beats the baseline.
Limitations
- Still requires a small amount of annotated training data (0.5%).
- Replanning relies on object similarity from language embeddings and is limited to objects observed so far.
- Robotic-arm experiment is small (65 pairs) and uses privileged low-level policies.
Experimental setup
Datasets: ALFRED
Benchmarks: ALFRED
Metrics: SR GC PLWSR PLWGC planner accuracy
Baselines: HLSM FILM CAPEAM LLM-Planner
Builds on: LLM-Planner FILM HLSM CLIP BERT
Closest papers by meaning
- Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation (AAAI 2025 · similarity 0.89)
- Embodied Task Planning via Graph-Informed Action Generation with Large Language Models (ICML 2026 · similarity 0.89)
- R2C: Mapping Room to Chessboard to Unlock LLM As Low-Level Action Planner (CVPR 2025 · similarity 0.88)
- Efficient Skill Grounding via Code Refactoring with Small Language Models (ICML 2026 · similarity 0.88)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.88)
- LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.