AI Paper Atlas

Research map / Large language models / AI and society

Evaluating Language Models' Evaluations of Games

Katherine M. Collins, Cedegao E. Zhang, Graham Todd, Lance Ying, Mauricio Barba da Costa, Ryan Liu, Prafull Sharma, Adrian Weller, Ionatan Kuperwajs, Lionel Wong, Joshua B. Tenenbaum, Thomas L. Griffiths

ICLR 2026 · Large language models › AI and society · approach: Social-science view of AI evaluation

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
allow chain-of-thought before answering instead of direct answers (ablation)non-reasoning LLMs (GPT-4, LLaMA 3.1 70B, DeepSeek v3) · payoff vs GTO, 78 gamesR2: GPT-4 0.31 -> 0.38Improved
use reasoning models instead of non-reasoning modelspayoff evaluation · payoff, 121 gamesR2 with human judgments: GPT-4 CoT 0.38 -> o3 0.71Improved
scale up sophistication from o3 to GPT-5OpenAI reasoning models · payoff evaluationfit to human judgments: fit to humans drops while R2 vs GTO 0.71 -> 0.82Hurt
increase sophistication from GPT-4 to o1 and o3OpenAI models · payoff evaluationfit to human and GTO: R2 vs GTO 0.38 -> 0.50 -> 0.71Improved
use reasoning models for funness evaluationfunness evaluation · funness, 121 gamesfit to human funnessMixed
vary reasoning effort low/medium/high (ablation)o3 and GPT-5 · payoff evaluationaggregate fit to human dataNo clear effect
increase reasoning effort from low to high (ablation)GPT-5 · payoff for 10x10 3 (P1 D) gamepredicted payoff: -0.74 -> 0.86 (human 0.04)Mixed
game novelty (traits differing from Tic-Tac-Toe) vs reasoning token usagereasoning models · 121 gamesreasoning tokensNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy