Research map / Multi-agent systems and games / LLM agents
PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PARTNR, a 100K-task natural-language benchmark for human-robot collaboration in simulated households, generated with LLMs plus simulation and human filtering, and evaluates LLM planners (ReAct, RAG, finetuned Llama3.1-8B) with simulated and real humans.
Problem
Existing embodied AI benchmarks either have robots act in isolation or lack natural-language tasks, and few evaluate collaboration between robots and humans on diverse, long-horizon household tasks with tailored evaluation functions.
Approach
A semi-automated pipeline where LLMs generate tasks and Python predicate-based evaluation functions, grounded via simulation-in-the-loop filtering and human annotation of 1,000 seed examples, then scaled to 100,000 tasks by in-context modification; evaluation covers constraint-free, spatial, temporal, and heterogeneous task types in multi-agent settings.
Key results
- Humans solve 93% of PARTNR tasks, while the best LLM planner without privileged information reaches only 0.30 success rate (ReAct with ConceptGraphs perception).
- Finetuned Llama3.1-8B achieves 0.70 success vs 0.73 for Llama3.1-70B ReAct while being 8.6x faster at inference.
- Decentralized ReAct needs 3295 simulation steps versus 2519 for single-agent ReAct, indicating a coordination burden; with real humans, two-human teams finish in 2369 steps versus 3046 for a single human.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use centralized rather than decentralized ReAct planning | ReAct with oracle skills and partial observability · PARTNR test | Sim Steps: 3295.20 { $\pm$ 76.27} -> 2298.13 { $\pm$ 61.39} | Helped |
| Add retrieval-augmented planning examples | Decentralized ReAct · PARTNR test | Success Rate: 0.73 { $\pm$ 0.01} -> 0.71 { $\pm$ 0.01} | Hurt |
| Replace the 70B planner with a finetuned 8B planner | Decentralized ReAct · PARTNR test | Success Rate: 0.73 { $\pm$ 0.01} -> 0.70 { $\pm$ 0.01} | No clear effect |
| Replace oracle skills with learned skills | Decentralized ReAct · PARTNR test | Success Rate: 0.73 -> 0.57 | Hurt |
| Replace privileged perception with ConceptGraph perception | Decentralized ReAct with learned skills · PARTNR test | Success Rate: 0.57 -> 0.30 | Hurt |
| Use two decentralized agents instead of one agent | ReAct with oracle skills and partial observability · PARTNR test | Sim Steps: 2519.02 { $\pm$ 57.48} -> 3295.20 { $\pm$ 76.27} | Hurt |
| Use two human collaborators rather than one | Human participants · PARTNR validation | Sim Steps: 3046.99 { $\pm$ 80.79} -> 2369.55 { $\pm$ 49.33} | Helped |
| Use a finetuned robot instead of a ReAct robot with a real human partner | Human-robot collaboration · PARTNR validation | Sim Steps: 4267.71 { $\pm$ 83.40} -> 3443.33 { $\pm$ 61.46} | Helped |
| Use a finetuned robot instead of a ReAct robot with a real human partner | Human-robot collaboration · PARTNR validation | Task Offloading: 16% -> 26% | Helped |
| Use a robot partner instead of completing the task alone | Human participant with ReAct or finetuned robot · PARTNR validation | Sim Steps: 3046.99 { $\pm$ 80.79}; 4267.71 { $\pm$ 83.40}; 3443.33 { $\pm$ 61.46} | Hurt |
Surprising findings
- Human-LLM teams are slower than a single human working alone, even though LLM partners offload about 26% of tasks.
- LLM baselines achieve 0.92 and 0.91 success with real humans, since humans adapt to LLM mistakes, versus 0.30 without humans.
Limitations
- Generation accuracy is about 83% combined (90% instructions times 92% evaluation functions), so some tasks remain erroneous.
- LLM planners rely on privileged information and struggle with skill failures and perception errors.
- Simulated human partners are LLM-controlled and cannot recover from partner mistakes.
Experimental setup
Datasets: HSSD OVMM object set
Benchmarks: PARTNR
Metrics: success rate percent complete simulation steps planning cycles task offloading extraneous effort
Baselines: ReAct ReAct-RAG Heuristic expert Finetuned Llama3.1-8B CoELA
Builds on: ReAct Habitat 3.0 HSSD ConceptGraphs retrieval-augmented generation
Closest papers by meaning
- LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents (ICLR 2024 · similarity 0.92)
- Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (NeurIPS 2024 · similarity 0.90)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
- LLawCo: Learning Laws of Cooperation for Modeling Embodied Multi-Agent Behavior (ICML 2026 · similarity 0.90)
- From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents (ICLR 2026 · similarity 0.89)
- Collaborative Tree Search for Enhancing Embodied Multi-Agent Collaboration (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.