Research map / Large language models / Agents, code and math
LLM Strategic Reasoning: Agentic Study through Behavioral Game Theory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a behavioral game-theory framework (truncated quantal response equilibrium, TQRE) that estimates an LLM's strategic reasoning depth τ from choice frequencies over 13 matrix games, applied to 22 LLMs with CoT and demographic persona analyses.
Problem
Existing evaluations of LLM strategic reasoning rely on Nash equilibrium binary checks, which ignore bounded rationality, reasoning depth, and the mechanisms behind model choices.
Approach
Fit a Poisson mixture of level-k logit (quantal) best-responses, as in cognitive hierarchy/QRE, to each LLM's empirical action frequencies over a library of matrix games to estimate a reasoning depth parameter τ and a precision parameter γ, then analyze reasoning traces, CoT effects, and demographic persona shifts.
Key results
- GPT-o1, GPT-o3-mini and DeepSeek-R1 show the highest reasoning depth τ across most games (GPT-o1 top in 6 games, GPT-o3-mini in 7, DeepSeek-R1 in 5).
- Longer reasoning chains do not consistently yield better decisions; GPT-o1 and DeepSeek-R1 produce the shortest CoT in their strongest games.
- Demographic personas shift reasoning depth significantly, e.g., female persona raises τ for GPT-4o, while homosexual/bisexual personas lower τ for Gemini 2.0 and Granite MoE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CoT prompting (ablation) | LLMs evaluated on strategic games · Multiple game types | reasoning depth (τ) | Mixed |
| Add CoT prompting (ablation) | Claude-3-Opus · Competitive games, BL | reasoning depth (τ): 1.131 -> 4.264 | Helped |
| Add CoT prompting (ablation) | GPT-4o · Competitive games, BL | reasoning depth (τ): 1.543 -> 0.511 | Hurt |
| Distill from DeepSeek-R1 (ablation) | LLaMA and Qwen models · Cooperative and mixed-motive games | reasoning depth (τ) | Helped |
| Assign a female persona | GPT-4o · Demographic persona evaluation | reasoning depth coefficient: 0.274* | Helped |
| Assign a female persona | Claude-3-Opus · Demographic persona evaluation | reasoning depth coefficient: 0.146** | Helped |
| Assign a homosexual persona | LLaMA-8B · Demographic persona evaluation | reasoning depth coefficient: -0.418* | Hurt |
| Assign demographic personas (ablation) | LLMs with CoT prompting · Demographic persona evaluation | number of demographic variables with significant influence | Hurt |
Surprising findings
- Higher token counts in internal reasoning do not correlate with better strategic reasoning.
- CoT prompting has mixed effects on strategic reasoning, and can impair models such as GPT-4o.
Limitations
- Causal link between prompts and decision outcomes remains underexplored.
- Framework is semi-dynamic and does not cover real-time multi-agent interaction.
- Reasoning-style analysis relies on qualitative coding of a limited set of traces.
Experimental setup
Datasets: 13 custom matrix games SW10 matrix (human reasoning benchmark)
Benchmarks: 13 matrix games library (complete and incomplete information)
Metrics: estimated reasoning depth tau precision gamma maximum log-likelihood vs chance baseline OLS demographic regression coefficients
Baselines: GPT-4o GPT-o1 GPT-o3-mini DeepSeek-R1 Claude-3-Opus Gemini-2.0
Builds on: quantal response equilibrium cognitive hierarchy chain-of-thought prompting
Closest papers by meaning
- GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.92)
- Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis (AAAI 2024 · similarity 0.89)
- Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Games (ICML 2026 · similarity 0.88)
- LLM Agents Can Be Choice-Supportive Biased Evaluators: An Empirical Study (AAAI 2025 · similarity 0.88)
- Sparks of Cooperative Reasoning: LLMs as Strategic Hanabi Agents (ICML 2026 · similarity 0.87)
- KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.