Research map / Multi-agent systems and games / LLM agents
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AgentBoard, a multi-turn, partially observable benchmark of 9 text-based environments with a fine-grained subgoal-based progress rate and an open-source analytical evaluation toolkit, used to evaluate 18 LLMs.
Problem
Evaluating LLMs as multi-turn agents in partially observable environments within a unified framework, where final success rate alone gives little insight into intermediate progress and failure modes.
Approach
A unified benchmark of 9 text-based agent environments with human-annotated subgoals enabling a progress rate metric (max matching score over subgoal completion), plus an analytical toolkit reporting grounding accuracy, easy/hard breakdowns, long-range progress, and sub-skill scores.
Key results
- GPT-4 achieves the highest average progress rate of 70.0% and success rate of 47.9% across the 9 environments.
- Progress rate separates similarly-low-success open models; e.g., Mistral-7b (24.6% progress, 3.9% success) vs Llama2-13b (18.9% progress, 2.1% success).
- Progress rate correlates with human progress judgments with Pearson above 0.95 on all tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace sliding-window history with cutoff (ablation) | GPT-3.5-Turbo agent · AgentBoard Avg. | progress rate: 41.4 -> 32.8 | Hurt |
| replace sliding-window history with cutoff (ablation) | GPT-3.5-Turbo agent · AgentBoard Avg. | success rate: 19.7 -> 13.7 | Hurt |
| replace sliding-window history with summary (ablation) | GPT-3.5-Turbo agent · AgentBoard Avg. | progress rate: 41.4 -> 33.5 | Hurt |
| replace sliding-window history with summary (ablation) | GPT-3.5-Turbo agent · AgentBoard Avg. | success rate: 19.7 -> 14.4 | Hurt |
| replace sliding-window history with cutoff or summary (ablation) | Mistral-7b agent · AgentBoard Avg. | progress rate: 24.6 -> 24.0; 24.6 -> 23.9 | No clear effect |
| replace Act with ReAct prompting (ablation) | GPT-3.5-Turbo agent · AlfWorld | success rate: 17.2 -> 8.2 | Hurt |
| replace Act with ReAct prompting (ablation) | GPT-3.5-Turbo agent · PDDL | progress rate: 25.0 -> 17.1 | Hurt |
| replace Act with ReAct prompting (ablation) | GPT-3.5-Turbo agent · Tool-Query | success rate: 45.0 -> 50.0 | Helped |
| replace Act with ReAct prompting (ablation) | GPT-3.5-Turbo agent · Tool-Operation | progress rate: 37.2 -> 54.8 | Helped |
| train Llama2-70b with agent instruction-tuning trajectories | Llama2-70b · AgentBoard Avg. | progress rate: +9.5% | Helped |
| train Llama2-70b with agent instruction-tuning trajectories | Llama2-70b · AgentBoard Avg. | success rate: +10.2% | Helped |
| increase model scale from Llama2-13b to Llama2-70b | Llama2 agent · AgentBoard Avg. | progress rate: 18.9 -> 23.8 | Helped |
Surprising findings
- Code-pretrained models (CodeLlama-34b, Lemur-70b) outperform general Llama2-70b, suggesting code data helps agentic abilities beyond code generation.
- Text-Davinci-003 has only 58.9% grounding accuracy yet performs comparably to GPT-3.5-Turbo in main results.
Limitations
- Subgoal annotation is human-dependent and not scalable; LLMs cannot yet generate accurate subgoals.
- Evaluation is limited to simulated environments rather than real-world tasks.
- Proprietary model results show larger confidence intervals due to decoding post-processing.
Experimental setup
Datasets: AlfWorld ScienceWorld BabyAI Jericho PDDL WebShop WebArena Tool-Query Tool-Operation
Benchmarks: AgentBoard AlfWorld ScienceWorld BabyAI Jericho PDDL WebShop WebArena Tool-Query Tool-Operation
Metrics: progress rate success rate grounding accuracy sub-skill score
Baselines: GPT-4 GPT-3.5-Turbo Claude2 Gemini1.5-Flash Llama3-70b xLAM-70b AgentLM-70b Lemur-70b
Builds on: AgentBench ReAct AlfWorld WebArena
Closest papers by meaning
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.92)
- Harbor Adapters and Harbor-Mix: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation (NeurIPS 2026 · similarity 0.91)
- PACE: A Proxy for Agentic Capability Evaluation (NeurIPS 2026 · similarity 0.90)
- AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.89)
- AgentSquare: Automatic LLM Agent Search in Modular Design Space (ICLR 2025 · similarity 0.89)
- Large Language Models Miss the Multi-agent Mark (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.