Research map / Multi-agent systems and games / LLM agents

Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Games

ICML 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Strat-Reasoner, an RL framework that trains a Qwen3-4B LLM with recursive opponent-aware reasoning, a judge-scored CoT alignment signal, and a hybrid advantage combining CoT and return advantages for strategic play in two-player games.

Problem

LLMs struggle in multi-agent games where outcomes depend on all agents' joint strategies; non-stationary opponents, sparse credit assignment across multi-turn reasoning, and lack of process-level signals hamper RL training.

Approach

A recursive "Past-Present-Future" reasoning template that makes the ego agent model the opponent's intent and predictions, plus a centralized CoT comparison (base model as judge scoring belief-vs-actual alignment) whose micro-rollout-normalized scores are added to return-based advantages in a GRPO-style objective.

Key results

• Strat-Reasoner-4B reaches 90.77 (first-move) on Tic-Tac-Toe vs 100-sim MCTS, versus 88.84 for GPT-5-mini and 76.53 for Qwen3-32B.
• On KuhnPoker vs NE bot it scores 94.04 (first-move), above Gemini-2.5-flash (92.44) and GPT-5-mini (77.72).
• On OOD games it scores 70.39 on ConnectFour (10 sims, first-move) vs 68.00 for SPIRAL-4B and 66.98 for MARSHAL-4B; the abstract claims 22.1% average improvement.

Surprising findings

• Unnormalized CoT scores as advantages cause training instability and performance degradation in MiniHanabi.
• Excluding CoT training signals prevents the agent from exceeding baseline levels in MiniHanabi.

Limitations

• Only two-player settings evaluated; N-agent extension left as future work.
• Relies on the base model as CoT judge, with scores validated on only one Tic-Tac-Toe example.
• Evaluation against static bots (MCTS, NE policies) rather than diverse learned opponents.

Experimental setup

Datasets: Tic-Tac-Toe (OpenSpiel) KuhnPoker (OpenSpiel) MiniHanabi (OpenSpiel)

Benchmarks: Tic-Tac-Toe vs MCTS KuhnPoker vs NE bot MiniHanabi co-op ConnectFour vs MCTS (OOD) LeducHoldem vs NE bot (OOD) SimpleHanabi (OOD)

Metrics: normalized average game score

Baselines: GPT-5-mini Gemini-2.5-flash Qwen3-8B Qwen3-32B Gemma3-12B SPIRAL-4B MARSHAL-4B

Builds on: GRPO MARSHAL SPIRAL CTDE Cognitive Hierarchy theory

Closest papers by meaning

• MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs (ICLR 2026 · similarity 0.89)
• GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.89)
• From Debate to Equilibrium: Belief‑Driven Multi‑Agent LLM Reasoning via Bayesian Nash Equilibrium (ICML 2025 · similarity 0.89)
• DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs (AAAI 2026 · similarity 0.89)
• Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Search (ICLR 2025 · similarity 0.89)
• Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.88)