Research map / Large language models / Agents, code and math
GameArena: Evaluating LLM Reasoning through Live Computer Games
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GameArena, a dynamic benchmark that evaluates LLM reasoning through three live human-played games (Akinator, Taboo, Bluffing) and retrospective analysis of step-by-step reasoning outputs.
Problem
Static reasoning benchmarks suffer from contamination and saturation, while Chatbot Arena's preference-based rankings are confounded by style and do not isolate specific reasoning capabilities.
Approach
Embed LLMs in rule-constrained human games so that game outcomes and retrospective intermediate outputs (ranked object/word lists, truthfulness predictions) yield fine-grained procedural metrics for deductive, abductive, inductive, and multi-hop reasoning.
Key results
- Claude 3.5 Sonnet had the highest Akinator recall rate (27.83%) and best Taboo abductive metrics, outperforming GPT-4o in most reasoning metrics.
- 86.9% of GameArena sessions were useful versus only 4% of Chatbot Arena conversations.
- User study with 100 participants: over 70% liked GameArena games versus 45% enjoying Chatbot Arena voting; over 80% reported satisfaction versus under 40% for Chatbot Arena.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GameArena's interactive Akinator evaluation | LLM evaluation · Akinator | useful data rate: 4% -> 86.9% | Helped |
| Use GameArena's interactive game evaluation | user experience versus Chatbot Arena · User study | user enjoyment: 45% -> over 70% | Helped |
| Use GameArena gameplay instead of Chatbot Arena voting | user experience · User study | user satisfaction: less than 40% -> over 80% | Helped |
| Evaluate reasoning with Claude 3.5 Sonnet rather than GPT-4o | GameArena · Akinator | recall rate (%): 20.32 -> 27.83 | Helped |
| Evaluate reasoning with GPT-4o rather than Claude 3.5 Sonnet | GameArena · Taboo | win rate: 0.61±0.18 -> 0.67±0.11 | Helped |
| Evaluate reasoning with Claude 3.5 Sonnet rather than Gemini-1.5-Pro | GameArena · Bluffing | win rate: 0.60±0.18 -> 0.67±0.13 | Helped |
| Use LLaMA-3.1-405B rather than Claude 3.5 Sonnet | GameArena retrospective analysis · Taboo | recall rate (%): 68.85 -> 71.22 | Helped |
| Use LLaMA-3.1-405B rather than Claude 3.5 Sonnet | GameArena retrospective analysis · Bluffing | recall rate (%): 14.12 -> 19.05 | Helped |
| Use retrospective deductive reasoning rankings rather than Akinator outcome rankings | GameArena ranking · LiveBench-Reasoning | Kendall's Tau: 0.6 -> 0.8 | Helped |
| Use retrospective abductive reasoning rankings rather than Taboo outcome rankings | GameArena ranking · LiveBench-Reasoning | Kendall's Tau: -0.2 -> 0.8 | Helped |
Surprising findings
- Mistral Large often failed to follow game rules and make final predictions, yielding near-zero win rates in Akinator and Bluffing.
- LLaMA-3.1 405B had the highest Taboo recall rate but weaker Akinator performance, which the authors attribute to passive versus active task differences.
Limitations
- Only three games and five models evaluated, with a small controlled-experiment subset of 50 sessions per set.
- Taboo outcome ranking disagrees strongly with other benchmarks because the game does not directly test hypothesis generation.
- Metrics rely on models' self-reported retrospective outputs, which may not faithfully reflect hidden reasoning.
Experimental setup
Datasets: GameArena gaming sessions collected with Cloud Research participants Chatbot Arena conversation data (one week sample)
Benchmarks: GameArena (Akinator, Taboo, Bluffing) Chatbot Arena LiveBench-Reasoning GPQA
Metrics: win rate average rounds recall rate top-5 recall rate top-10 recall rate disparity ratio average first appear round average final rank Spearman's coefficient hopping penalty
Baselines: Chatbot Arena LiveBench-Reasoning GPQA
Builds on: Chatbot Arena DSPy Games with a purpose
Closest papers by meaning
- KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation (NeurIPS 2025 · similarity 0.91)
- GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.90)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.90)
- SmartPlay : A Benchmark for LLMs as Intelligent Agents (ICLR 2024 · similarity 0.89)
- Large Language Models Play StarCraft II:Benchmarks and A Chain of Summarization Approach (NeurIPS 2024 · similarity 0.89)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.