Research map / Large language models / AI and society
Evaluating Language Models' Evaluations of Games
Katherine M. Collins, Cedegao E. Zhang, Graham Todd, Lance Ying, Mauricio Barba da Costa, Ryan Liu, Prafull Sharma, Adrian Weller, Ionatan Kuperwajs, Lionel Wong, Joshua B. Tenenbaum, Thomas L. Griffiths
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| allow chain-of-thought before answering instead of direct answers (ablation) | non-reasoning LLMs (GPT-4, LLaMA 3.1 70B, DeepSeek v3) · payoff vs GTO, 78 games | R2: GPT-4 0.31 -> 0.38 | Improved |
| use reasoning models instead of non-reasoning models | payoff evaluation · payoff, 121 games | R2 with human judgments: GPT-4 CoT 0.38 -> o3 0.71 | Improved |
| scale up sophistication from o3 to GPT-5 | OpenAI reasoning models · payoff evaluation | fit to human judgments: fit to humans drops while R2 vs GTO 0.71 -> 0.82 | Hurt |
| increase sophistication from GPT-4 to o1 and o3 | OpenAI models · payoff evaluation | fit to human and GTO: R2 vs GTO 0.38 -> 0.50 -> 0.71 | Improved |
| use reasoning models for funness evaluation | funness evaluation · funness, 121 games | fit to human funness | Mixed |
| vary reasoning effort low/medium/high (ablation) | o3 and GPT-5 · payoff evaluation | aggregate fit to human data | No clear effect |
| increase reasoning effort from low to high (ablation) | GPT-5 · payoff for 10x10 3 (P1 D) game | predicted payoff: -0.74 -> 0.86 (human 0.04) | Mixed |
| game novelty (traits differing from Tic-Tac-Toe) vs reasoning token usage | reasoning models · 121 games | reasoning tokens | No clear effect |
Closest papers by meaning
- GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.90)
- Mastering Board Games by External and Internal Planning with Language Models (ICML 2025 · similarity 0.89)
- GameArena: Evaluating LLM Reasoning through Live Computer Games (ICLR 2025 · similarity 0.89)
- Improving Rationality in the Reasoning Process of Language Models through Self-playing Game (ICML 2025 · similarity 0.88)
- LLM Strategic Reasoning: Agentic Study through Behavioral Game Theory (NeurIPS 2025 · similarity 0.87)
- Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
