Research map / Large language models / Agents, code and math
Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Embodied Agent Interface, an LTL-based, object-centric framework that formalizes four LLM ability modules (goal interpretation, subgoal decomposition, action sequencing, transition modeling) and fine-grained error metrics, and benchmarks 18 LLMs on BEHAVIOR and VirtualHome.
Problem
Existing evaluations of LLMs for embodied decision making are fragmented across domains, input/output specifications, and rely mainly on a single final success rate, which makes it hard to locate what abilities are missing.
Approach
Unify embodied tasks with LTL goals over object-centric states and actions, define four standardized LLM ability modules with fixed input-output specs, and evaluate each module in isolation with fine-grained error categories (hallucination, affordance, missing/additional steps, wrong order) and F1/planner-based metrics.
Key results
- o1-preview achieves 74.9% average module success on BEHAVIOR versus 64.2% for the next best model (Claude-3.5 Sonnet).
- Trajectory feasibility errors are common (45.2%) for the evaluated models, with missing-step (19.5%) and additional-step (14.2%) errors being the largest categories.
- Relational goals and non-spatial relations are harder than object-state goals; non-spatial relation F1 for o1-preview is only 11.8% on VirtualHome transition modeling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use o1-preview instead of Claude-3.5 Sonnet | LLM embodied decision-making evaluation · BEHAVIOR | Module SR: 64.2% to 74.9% | Helped |
| Use replanning based on feedback | GPT-4o action sequencing · BEHAVIOR | Task SR: 47.0% to 59.0% | Helped |
| Use replanning based on feedback | GPT-4o action sequencing · BEHAVIOR | Execution SR: 53.0% to 63.0% | Helped |
| Use replanning based on feedback | GPT-4o action sequencing · BEHAVIOR | Additional step error rate: 0.0% to 3.0% | Hurt |
| Compose goal interpretation with subgoal decomposition in a pipeline | GPT-4o · BEHAVIOR | Task SR: 48.0% to 38.0% | Hurt |
| Use the pipeline-based rather than modularized goal interpretation and action sequencing | GPT-4o · BEHAVIOR | Execution SR: 53.0% to 55.0% | Helped |
| Increase task goal count from fewer than 5 to more than 10 | LLM action sequencing · BEHAVIOR | Task SR: around 60% ... below 40% | Hurt |
| Increase trajectory sequence length | LLM trajectory evaluation · BEHAVIOR and VirtualHome | Trajectory feasibility | Hurt |
| Increase environment complexity and variety of object and state features | LLM goal evaluation · BEHAVIOR and VirtualHome | Goal evaluation performance | Hurt |
| Evaluate spatial relation goals rather than object-state goals | LLM goal interpretation · BEHAVIOR and VirtualHome | Recall | Hurt |
Surprising findings
- Subgoal decomposition is not strictly easier than action sequencing in abstract action spaces.
- Mistral Large and Gemini 1.5 Pro outperform o1-preview on VirtualHome action sequencing task success.
Limitations
- Evaluation limited to abstract, language-described states, actions and goals; no sensory (vision) inputs.
- Does not cover memory, geometric reasoning, or navigation.
- Only two simulators (BEHAVIOR and VirtualHome) and a restricted LTL fragment (no globally or until).
Experimental setup
Datasets: BEHAVIOR VirtualHome
Benchmarks: BEHAVIOR VirtualHome
Metrics: F1 task success rate execution success rate planner success rate error-type rates
Baselines: LOTA-Bench
Builds on: PDDL linear temporal logic LLM+P Code as Policies SayCan
Closest papers by meaning
- From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents (ICLR 2026 · similarity 0.91)
- Embodied CoT Distillation From LLM To Off-the-shelf Agents (ICML 2024 · similarity 0.91)
- LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents (ICLR 2024 · similarity 0.91)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.91)
- Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning (NeurIPS 2025 · similarity 0.90)
- PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.