Research map / Large language models / Agents, code and math
LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoTa-Bench, an automatic benchmark for LLM-based language-oriented task planners for home-service embodied agents, using ALFRED/AI2-THOR and an extended WAH-NL/VirtualHome setup, and benchmarks many LLMs with extensions like example selection, replanning, and fine-tuning.
Problem
Comparing LLM-based task planners for embodied home agents is difficult because evaluation relies on human judgment or custom environments, and factors like model choice, in-context examples, replanning, and fine-tuning are under-explored.
Approach
An automatic simulator-based benchmark pairing a skill-scoring LLM planner (greedy constrained decoding via Guidance) with ALFRED/AI2-THOR and a crowdsourced-instruction extension of Watch-And-Help/VirtualHome, scoring success by comparing final simulator state to goal conditions.
Key results
- GPT-3 (text-davinci-003) achieved 21.36% task success on ALFRED and 40.82% average subgoal success on WAH-NL, the best among baselines in the main sweep.
- Semantic similarity example selection let LLaMA 1 65B reach 43.25% subgoal success on WAH-NL, surpassing the GPT-3 baseline's 40.82%.
- LoRA fine-tuning of LLaMA 1 30B on ALFRED raised success from 13.66% to 60.08%, but the ALFRED-tuned planner dropped to 10.38% on WAH-NL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GPT-4 instead of GPT-3 | LLM task planner · ALFRED | task success rate: 21.36% -> 40.38% | Helped |
| use GPT-4 instead of GPT-3 | LLM task planner · WAH-NL | average subgoal success rate: 40.82% -> 34.17% | Hurt |
| add failure feedback and replanning (ablation) | LLaMA 2 13B planner · ALFRED | task success rate: 21.15 (44/208) -> 17.79 (37/208) | Hurt |
| add failure feedback and replanning (ablation) | LLaMA 2 70B planner · ALFRED | task success rate: 21.63 (45/208) -> 24.04 (50/208) | Helped |
| fine-tune on ALFRED train set with LoRA (ablation) | LLaMA 1 30B planner · ALFRED valid-seen | task success rate: 13.66% -> 60.08% | Helped |
| transfer ALFRED fine-tuning to WAH-NL (ablation) | LLaMA 1 planner · WAH-NL | average subgoal success rate: 32.22% -> 10.38% | Hurt |
| select in-context examples by semantic similarity rather than randomly (ablation) | LLM task planner · WAH-NL | average subgoal success rate | Helped |
| increase the number of in-context examples (ablation) | LLaMA 2 13B planner · ALFRED and WAH-NL | success rate | Helped |
| use Guidance constrained skill selection instead of SayCan-style individual scoring (ablation) | GPT-Neo 6B planner · WAH-NL | average subgoal success rate: 21.65% -> 24.80% | Helped |
| use Guidance constrained skill selection instead of SayCan-style individual scoring (ablation) | GPT-Neo 6B planner · WAH-NL | processing time: 1649.2 minutes -> 87.4 minutes; about 19x speedup | Helped |
| use instruction- or chat-tuned models instead of base models (ablation) | baseline task planner · ALFRED and WAH-NL | success rate | Hurt |
| provide in-context examples to an already fine-tuned planner (ablation) | LLaMA 1 30B fine-tuned planner · ALFRED valid-unseen | task success rate: 44.59% | Hurt |
Surprising findings
- Smaller models sometimes outperformed larger ones (e.g., GPT-J 6B beat GPT-NeoX 20B; OPT 2.7B beat OPT 6.7B on WAH-NL).
- GPT-4 scored higher than GPT-3 on ALFRED (40.38%) but lower on WAH-NL (34.17% vs 40.82%).
Limitations
- High-level planning is decoupled from low-level actions and visual perception, so no end-to-end evaluation.
- Simulator-to-real domain gap; AI2-THOR and VirtualHome lack real-world diversity.
- Object instance recognition is not handled, so ALFRED Pick Two & Place was excluded and WAH-NL object counts were set to 1.
Experimental setup
Datasets: ALFRED Watch-And-Help WAH-NL
Benchmarks: ALFRED WAH-NL
Metrics: task success rate average subgoal success rate
Baselines: GPT-3 text-davinci-003 GPT-4 GPT-3.5-Turbo GPT-J 6B GPT-NeoX 20B OPT MPT LLaMA 2
Builds on: SayCan ALFRED Watch-And-Help Guidance LoRA
Closest papers by meaning
- PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks (ICLR 2025 · similarity 0.92)
- Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (NeurIPS 2024 · similarity 0.91)
- TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.91)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
- Embodied Task Planning via Graph-Informed Action Generation with Large Language Models (ICML 2026 · similarity 0.89)
- Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.