Research map / Multi-agent systems and games / LLM agents

AgentBench: Evaluating LLMs as Agents

ICLR 2024 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 53 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AgentBench, a multi-environment benchmark (8 interactive environments across code, game, and web groundings) for evaluating LLMs as agents, and evaluates 29 API-based and open-source LLMs with a unified HTTP-based toolkit.

Problem

There is no systematic, standard benchmark for evaluating LLMs as autonomous agents in interactive, multi-round environments; existing agent benchmarks focus on single environments or multimodal simulators.

Approach

A multi-dimensional benchmark of eight text-only interactive environments (OS, DB, KG, DCG, LTP, HH, WS, WB) with a normalized overall score weighting each task by the inverse of its average score across models, plus a decoupled server-client evaluation toolkit that uses max-flow scheduling to run agent-task evaluations.

Key results

• gpt-4 achieves an overall score of 4.01, best on 6 of 8 environments, and 78% success rate on House Holding.
• Best open-source model under 70B, codellama-34b, scores 0.96 overall versus an average of 0.51 for OSS models and 2.32 for API models.
• Vicuna-13b (0.93) outperforms llama-2-13b (0.77) on the same base model, suggesting high-quality alignment data helps agent performance.

Measured effects reported in the paper

Surprising findings

• llama-2-13b and llama-2-70b perform similarly across tasks despite the large size gap, attributed to insufficient pre-training or alignment of the 70B model.
• Code training has ambivalent effects: codellama improves Web Shopping but underperforms llama-2 on Digital Card Game and Operating System.

Limitations

• Evaluation uses only zero-shot-style CoT prompting without reflection, search, or multiple trials, so results reflect a primitive strategy.
• Open-source models are limited to 70B and smaller.
• Test sets are small for some tasks (e.g., 20 to 50 samples in DCG and LTP), increasing variance.

Experimental setup

Datasets: WikiSQL WikiTableQuestions SQA HybridaQA FeTaQA GrailQA ComplexWebQuestions GraphQuestions WebShop Mind2Web ALFWorld

Benchmarks: AgentBench ALFWorld WebShop Mind2Web

Metrics: success rate F1 win rate reward game progress step success rate

Baselines: gpt-4 gpt-3.5-turbo claude-2 llama-2-70b codellama-34b vicuna-13b

Builds on: ReAct Chain-of-Thought ALFWorld WebShop Mind2Web

Closest papers by meaning

• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (NeurIPS 2024 · similarity 0.92)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.91)
• AgentSquare: Automatic LLM Agent Search in Modular Design Space (ICLR 2025 · similarity 0.91)
• \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.91)
• TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.90)
• Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.90)