Research map / Multi-agent systems and games / Games and equilibria
Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolf
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes One Night Ultimate Werewolf as a multi-phase extensive-form Bayesian game with PBE derivations, then trains an RL discussion-tactic policy (CQL on GPT-4 game logs) that instructs an LLM agent to discuss strategically.
Problem
LLM-based agents in communication games like Werewolf neglect control over discussion tactics, which matter greatly under hidden roles and role-switching uncertainty in One Night Ultimate Werewolf (ONUW).
Approach
Formulate ONUW as a multi-phase extensive-form Bayesian game, prove PBEs for a three-player case with and without discussion, and use an offline RL (CQL) discussion policy over six discretized tactics (honest/deceptive × evidence/accusation/defense) to choose the tactic that LLM agents follow when speaking.
Key results
- In the three-player ONUW game, RL-instructed agents achieve lower NashConv than ReAct and LLM-instructed variants, with the RL discussion policy outperforming direct LLM prompting.
- In the five-player easy and hard settings, the RL-instructed agent achieves the highest Team Village win rate against the same agent versions across both settings (Gemini backend).
- In the standard five-player game as Player 3, the RL-instructed agent outperforms other versions against Gemini- and GPT-4-based agents, sometimes tying GPT-4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an RL-trained discussion policy | LLM-based agents · Three-player ONUW | NashConv | Helped |
| Train the discussion policy with RL instead of prompting the LLM to choose tactics (ablation) | LLM-based agents · Three-player ONUW | NashConv | Helped |
| Add an RL-trained discussion policy | LLM-based agents · Five-player ONUW, easy and hard settings | Team Village win rate | Helped |
| Add an RL-trained discussion policy | LLM-based agents playing as Player 3 · Standard five-player ONUW | Win rate | Helped |
| Add an RL-trained discussion policy | LLM-based agents playing as Player 3 · Standard five-player ONUW | Average votes | Helped |
Surprising findings
- GPT-4-based agents showed higher NashConv than Gemini-based agents because Gemini Werewolves more often both voted for the Robber.
- Performance gains of the discussion policy are smaller for Team Werewolf than for Team Village, attributed to Werewolves' inherently high win rates in the game mechanics.
Limitations
- Discretization of discussion tactics into six manually defined categories relies on game-specific characteristics
- Theoretical analysis covers only a three-player version with two Werewolves and one Robber under simplifying belief assumptions
Experimental setup
Datasets: GPT-4-generated ONUW game logs (five-player)
Benchmarks: three-player ONUW (NashConv) five-player ONUW easy and hard settings (win rates) five-player ONUW standard setting as Player 3 (win rates, average votes)
Metrics: NashConv win rate average votes
Baselines: ReAct ReAct (w.o. discussion) Belief agent (no discussion policy) Random discussion policy LLM-instructed discussion policy
Builds on: ReAct Conservative Q-Learning ChatArena
Closest papers by meaning
- Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization (ICML 2025 · similarity 0.90)
- Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game (ICML 2024 · similarity 0.90)
- Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game (NeurIPS 2025 · similarity 0.89)
- Improving Cooperation in Language Games with Bayesian Inference and the Cognitive Hierarchy (AAAI 2025 · similarity 0.86)
- Scaling Inference-Time Computation via Opponent Simulation: Enabling Online Strategic Adaptation in Repeated Negotiation (ICML 2026 · similarity 0.85)
- LLM Strategic Reasoning: Agentic Study through Behavioral Game Theory (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.