Research map / Large language models / Agents, code and math
SmartPlay : A Benchmark for LLMs as Intelligent Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SmartPlay, a benchmark of six games (bandits, Rock-Paper-Scissors, Tower of Hanoi, MESSENGER, Crafter, Minecraft) that evaluates LLM agents on nine capabilities via text observations, and compares GPT-4 variants and other LLMs against a human baseline.
Problem
There is no systematic benchmark for evaluating LLMs as interactive agents across capabilities such as planning, understanding randomness, spatial reasoning, learning from interaction, and error handling.
Approach
Build a unified, text-based OpenAI-Gym-style interface over six games with varying difficulty degrees for nine agent capabilities, enabling per-capability analysis of LLM agents against a human baseline.
Key results
- GPT-4 variants score about 0.26 (GPT-4-0613) and 0.32 (GPT-4-0314) normalized on Crafter, versus 1.00 for the human baseline, with a roughly 70% gap.
- Best LLMs reach about 0.61 of human baseline on the Minecraft creative find task, and all LLMs perform similarly there, suggesting a 3D spatial reasoning bottleneck.
- GPT-4 variants score 0.83-0.90 on Tower of Hanoi (3-disk) and near-human on Bandit (1.00), while open-source 13B models reach at most about 0.5 on Bandit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4-0314 instead of GPT-4-0613 | LLM agents · RPS | normalized average score: 0.91 -> 0.98 | Helped |
| Use GPT-4-0314 instead of GPT-4-0613 | LLM agents · Hanoi | normalized average score: 0.83 -> 0.90 | Helped |
| Use GPT-4-0314 instead of GPT-4-0613 | LLM agents · Crafter | normalized average score: 0.26 -> 0.32 | Helped |
| Use GPT-4-0314 instead of GPT-4-0613 | LLM agents · Bandit | normalized average score: 1.00 -> 0.97 | Hurt |
| Use GPT-4-0314 instead of GPT-4-0613 | LLM agents · MessengerL1 | normalized average score: 0.90 -> 0.87 | Hurt |
| Use GPT-4-0314 instead of GPT-4-0613 | LLM agents · MessengerL2 | normalized average score: 0.93 -> 0.97 | Hurt |
| Use GPT-4-0314 instead of GPT-4-0613 | LLM agents · Minecraft | normalized average score: 0.61 -> 0.59 | No clear effect |
| Fine-tune LLAMA-13b into Vicuna-13b | open-source LLM agents · Hanoi | normalized average score: 0.33 -> 0.07 | Hurt |
| Use GPT-4-0314 instead of Human Baseline | SmartPlay agent · Crafter | normalized average score: 1.00 -> 0.32 | Hurt |
| Use GPT-4-0613 instead of Human Baseline | SmartPlay agent · Minecraft | normalized average score: 1.00 -> 0.61 | Hurt |
Surprising findings
- LLMs could write the Tower of Hanoi solution from the starting configuration but got confused after a few moves once disks spread across rods, suggesting contamination-resistance from intermediate states rarely seen in training.
- Vicuna-13b, the fine-tuned model, performs much worse than its base LLaMA-13b on many capabilities.
Limitations
- Only 7 of the settings were evaluated to reduce query cost, and human baseline uses only 3 players.
- Minecraft is simplified to text descriptions without vision, and only creative find tasks are included.
Experimental setup
Datasets: MESSENGER Crafter MineDojo Two-armed bandits (gym-bandits) Rock Paper Scissors Tower of Hanoi (gym-hanoi)
Benchmarks: SmartPlay
Metrics: reward completion rate score human-normalized score
Baselines: GPT-4-0613 GPT-4-0314 text-davinci-003 Claude Bard llama-2-13b llama-13b vicuna-13b human baseline
Builds on: MESSENGER Crafter MineDojo OpenAI Gym
Closest papers by meaning
- GameArena: Evaluating LLM Reasoning through Live Computer Games (ICLR 2025 · similarity 0.89)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.89)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.88)
- GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.88)
- MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs (ICLR 2026 · similarity 0.88)
- TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.