Research map / Multi-agent systems and games / Human-AI and goal-conditioned
ThinkBot: Embodied Instruction Following with Thought Chain Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ThinkBot, which uses GPT-3.5 as an instruction completer to recover missing actions and interacted objects from sparse instructions, plus a multimodal transformer object localizer, achieving state-of-the-art success on ALFRED.
Problem
Embodied instruction following on ALFRED fails because sparse, step-wise human instructions are incoherent, omitting intermediate actions (e.g., opening a fridge) and interacted objects needed for execution.
Approach
Reason over the thought chain in sparse instructions with an LLM (GPT-3.5-turbo prompted with scene-object and subgoal context) to recover missing actions and objects, then use a BERT-and-map multimodal transformer with a learned object correlation graph to localize the interacted object on the semantic map.
Key results
- On ALFRED test unseen, ThinkBot reaches 57.82% SR versus 55.46% for Prompter+ and 49.84% for CPEM.
- On ALFRED test seen, ThinkBot achieves 62.69% SR and 71.64% GC, outperforming Prompter+ (60.86% SR).
- On the hard valid unseen split, removing the instruction completer drops SR to 0%, while full ThinkBot reaches 22.97% SR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ThinkBot instead of Prompter+ | ALFRED EIF agent · ALFRED test seen | SR: 60.86 -> 62.69 | Helped |
| Use ThinkBot instead of Prompter+ | ALFRED EIF agent · ALFRED test unseen | SR: 55.46 -> 57.82 | Helped |
| Add the instruction completer (ablation) | ThinkBot · ALFRED valid unseen | SR: 64.92 -> 67.72 | Helped |
| Add the instruction completer (ablation) | ThinkBot · ALFRED hard valid unseen | SR: 0 -> 22.97 | Helped |
| Add the object localizer (ablation) | ThinkBot · ALFRED valid unseen | SR: 66.99 -> 67.72 | Helped |
| Add the object localizer (ablation) | ThinkBot · ALFRED hard valid unseen | SR: 16.22 -> 22.97 | Helped |
| Add the object correlation graph (ablation) | ThinkBot object localizer · ALFRED valid unseen | SR: 67.36 -> 67.72 | Helped |
| Add the object correlation graph (ablation) | ThinkBot object localizer · ALFRED hard valid unseen | SR: 21.62 -> 22.97 | Helped |
Surprising findings
- Without the instruction completer, SR on the hard valid unseen split drops to zero, showing that sparse instructions alone cannot handle closed-container tasks.
Limitations
- Relies on a closed API LLM (GPT-3.5-turbo), so reproducibility and cost depend on external service.
- Evaluated only on ALFRED simulation; no real-world validation.
- Hard-case success remains low (22.97% SR on hard valid unseen).
Experimental setup
Datasets: ALFRED
Benchmarks: ALFRED
Metrics: success rate (SR) goal-condition success rate (GC) path-length-weighted SR (PLWSR) path-length-weighted GC (PLWGC)
Baselines: Seq2seq MOCA E.T. LWIT HITUT ABP LLM-Planner FILM LGS-RPA Prompter CPEM Prompter+
Builds on: Prompter FILM CPEM ALFRED BERT
Closest papers by meaning
- Infer Human’s Intentions Before Following Natural Language Instructions (AAAI 2025 · similarity 0.88)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.88)
- ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning (NeurIPS 2025 · similarity 0.88)
- Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Following (NeurIPS 2024 · similarity 0.87)
- OntoPlan: An Ontology-Grounded Scene Representation and Agentic Framework for Scalable Robot Task Planning (NeurIPS 2026 · similarity 0.87)
- Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.