Research map / Multi-agent systems and games / LLM agents
Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes strategic language agents for Werewolf that combine LLM hidden-role deduction and diverse language action candidate generation with a population-based MAPPO-trained RL policy that selects among candidates.
Problem
Pure LLM-based agents in multi-agent social deduction games like Werewolf show intrinsic bias in their action choices and struggle to deduce hidden roles from deceptive communication, leading to suboptimal strategic play.
Approach
Use an LLM for hidden-role deduction and to generate a diverse set of free-form language action candidates, then train a small self-attention RL policy over those candidates (embedded via text-embedding-ada-002) against a population of LLM agents with predefined play styles, so the policy corrects the LLM's biased action distribution.
Key results
- In round-robin tournament of 100 games per pair, the strategic language agent achieves the highest win rate as Villagers (0.30) and as Werewolves (0.70) against ReAct, ReCon, Concurrent, and Atomic agents.
- In human experiments with 160 recruited players, the agent's overall win rate with six human players (0.52) is comparable to humans (0.52).
- Removing the RL policy drops Villager win rate from 0.30 to 0.23 and Werewolf win rate from 0.70 to 0.61 in ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the RL policy to choose from diverse action candidates (ablation) | Strategic language agent · Werewolf, as Villagers | win rate: 0.23 (0.03) -> 0.30 (0.03) | Helped |
| Add the RL policy to choose from diverse action candidates (ablation) | Strategic language agent · Werewolf, as Werewolves | win rate: 0.61 (0.04) -> 0.70 (0.03) | Helped |
| Add diverse action generation (ablation) | Strategic language agent · Werewolf, as Villagers | win rate: 0.16 (0.02) -> 0.22 (0.04) | Helped |
| Add diverse action generation (ablation) | Strategic language agent · Werewolf, as Werewolves | win rate: 0.54 (0.04) -> 0.61 (0.05) | Helped |
| Add hidden role deduction (ablation) | Strategic language agent · Werewolf, as Villagers | win rate: 0.16 (0.02) -> 0.30 (0.03) | Helped |
| Add hidden role deduction (ablation) | Strategic language agent · Werewolf, as Werewolves | win rate: 0.54 (0.04) -> 0.70 (0.03) | Helped |
| Add evidence to hidden role deduction (ablation) | Strategic language agent · Werewolf, as Villagers | win rate: 0.28 (0.04) -> 0.30 (0.03) | Helped |
| Add reasoning to hidden role deduction (ablation) | Strategic language agent · Werewolf, as Villagers | win rate: 0.25 (0.03) -> 0.30 (0.03) | Helped |
| Add reliability to hidden role deduction (ablation) | Strategic language agent · Werewolf, as Villagers | win rate: 0.23 (0.02) -> 0.30 (0.03) | Helped |
| Use population-based training instead of self-play (ablation) | Strategic language agent · Werewolf, as Villagers | win rate: 0.26 (0.04) -> 0.30 (0.03) | Helped |
| Use population-based training instead of self-play (ablation) | Strategic language agent · Werewolf, as Werewolves | win rate: 0.66 (0.02) -> 0.70 (0.03) | Helped |
| Use 3 instead of 2, 4, or 5 action candidates (ablation) | Strategic language agent · Increased diversity | increased diversity: 0.35; 0.37; 0.23; 0.16 | Mixed |
Surprising findings
- Using the LLM to choose among diverse action candidates (without the RL policy) gives win rates comparable to removing diversity, suggesting the LLM still carries intrinsic bias when selecting actions.
- The pure LLM agent's first-night Doctor policy saves others with probability 0.38, while the optimal is self-saving.
Limitations
- Werewolf is a pure text environment without tone or facial cues.
- Relies on proprietary GPT-4 and OpenAI embedding API, with substantial API cost and throughput needs.
- Baselines had to be adapted to this environment, since few existing agents work out of the box for Werewolf.
Experimental setup
Datasets: self-generated Werewolf game rollouts
Benchmarks: text-based 7-player Werewolf environment
Metrics: win rate action distribution
Baselines: ReAct ReCon Concurrent agent Atomic agent (Cicero-inspired)
Builds on: ReAct Cicero MAPPO policy-space response oracles Tree-of-Thought
Closest papers by meaning
- Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization (ICML 2025 · similarity 0.94)
- Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game (NeurIPS 2025 · similarity 0.93)
- Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolf (NeurIPS 2024 · similarity 0.90)
- MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs (ICLR 2026 · similarity 0.88)
- Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.87)
- Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Search (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.