Research map / Large language models / Agents, code and math
GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GTBench, a language-driven environment of 10 game-theoretic tasks built on OpenSpiel, and evaluates LLM strategic reasoning against MCTS/TfT/random agents and in LLM-vs-LLM Elo competitions.
Problem
Existing LLM agent evaluations rely on role-playing or verbal games that mix in narrative complexity, masking whether LLMs have concrete strategic and logical reasoning ability; a clean, game-theoretic evaluation is lacking.
Approach
A unified language-driven game environment spanning a game-theoretic taxonomy (complete/incomplete information, dynamic/static, probabilistic/deterministic), scored with a Normalized Relative Advantage metric and Elo ratings, with LLM-vs-LLM competition as an automated, adaptive evaluation.
Key results
- LLMs almost always fail (NRA of -1) against MCTS agents in complete deterministic games such as Tic-Tac-Toe, Connect-4, Breakthrough and Nim.
- Llama-3-70b-Instruct reaches an average Elo of 1577.06, close to GPT-4's 1591.99, and outperforms GPT-3.5-turbo (1514.30).
- CodeLlama-34b-Instruct (avg NRA -0.01) outperforms the larger Llama-2-70b-chat (avg NRA -0.20) and is comparable to GPT-3.5-turbo; advanced reasoning (CoT, ToT) does not consistently help.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use code-pretrained CodeLlama-34b-Instruct instead of Llama-2-70b-chat | LLM-vs.-LLM competition · Average across games | avg. NRA: -0.20 -> -0.01 | Helped |
| Use code-pretrained Deepseek-Coder-6.7b-instruct instead of Deepseek-LLM-67b-chat | LLM-vs.-LLM competition · Average across games | avg. NRA: -0.05 -> -0.03 | Helped |
| Add CoT | GPT-3.5-turbo · Against GPT-3.5-turbo with Prompt Agent | avg. NRA: 0.00 -> 0.02 | Helped |
| Add CoT | Llama-3-70b-Instruct · Against GPT-3.5-turbo with Prompt Agent | avg. NRA: 0.04 -> 0.07 | Helped |
| Add CoT | GPT-4 · Against GPT-3.5-turbo with Prompt Agent | avg. NRA: 0.13 -> 0.13 | No clear effect |
| Add CoT | CodeLlama-34b-Instruct · Against GPT-3.5-turbo with Prompt Agent | avg. NRA: -0.01 -> -0.09 | Hurt |
| Add CoT | CodeLlama-34b-Instruct · Against GPT-4 with Prompt Agent | avg. NRA: -0.01 -> -0.04 | Hurt |
| Add CoT | Llama-2-70b-chat · Against GPT-4 with Prompt Agent | avg. NRA: -0.10 -> -0.23 | Hurt |
| Remove Breakthrough from the evaluated game set (ablation) | GPT-4 · Average across games | avg. NRA: -0.02 | Hurt |
| Remove Breakthrough from the evaluated game set (ablation) | Llama-3-70b-Instruct · Average across games | avg. NRA: -0.05 | Hurt |
| Remove Breakthrough from the evaluated game set (ablation) | CodeLlama-34b-Instruct · Average across games | avg. NRA: +0.09 | Helped |
| Remove Breakthrough from the evaluated game set (ablation) | Llama-2-70b-chat · Average across games | avg. NRA: +0.02 | Helped |
Surprising findings
- Code-pretrained models with fewer parameters (CodeLlama-34b) outperform larger chat models such as Llama-2-70b-chat and Deepseek-LLM-67b-chat.
- Advanced reasoning methods such as CoT often lead to worse NRA for most LLMs, with only some benefit for powerful models.
Limitations
- Results may vary due to prompt sensitivity from pre-defined templates.
- Game action/state spaces are limited, so they may not separate LLMs of similar skill levels.
- Generated actions may be illegal due to instruction-following failures.
Experimental setup
Datasets: OpenSpiel game environments
Benchmarks: GTBench
Metrics: Normalized Relative Advantage Elo rating task completion rate regret Pareto efficiency
Baselines: Monte Carlo Tree Search agent Tit-for-Tat Random agent GPT-3.5-turbo with Prompt Agent
Builds on: OpenSpiel Chain-of-Thought Tree-of-Thought Self-Consistency
Closest papers by meaning
- LLM Strategic Reasoning: Agentic Study through Behavioral Game Theory (NeurIPS 2025 · similarity 0.92)
- KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation (NeurIPS 2025 · similarity 0.91)
- GameArena: Evaluating LLM Reasoning through Live Computer Games (ICLR 2025 · similarity 0.90)
- Sparks of Cooperative Reasoning: LLMs as Strategic Hanabi Agents (ICML 2026 · similarity 0.90)
- Evaluating LLMs in Open-Source Games (NeurIPS 2025 · similarity 0.90)
- Large Language Models Play StarCraft II:Benchmarks and A Chain of Summarization Approach (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.