Research map / Multi-agent systems and games / LLM agents
Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LSPO, an iterative loop that clusters LLM-generated Werewolf discussion utterances into a finite latent strategy space, solves the abstracted game with Deep CFR, and aligns the LLM to the resulting policy with DPO, improving Werewolf win rates over four baselines.
Problem
LLM agents in free-form strategic language games like Werewolf suffer from intrinsic bias in action distributions and limited exploration of the unbounded text action space, which makes game-theoretic solvers computationally infeasible on raw language actions.
Approach
Cluster free-form discussion utterances into a finite set of latent strategies via embeddings and k-means, reformulate the game as an extensive-form game over these latent strategies, solve it with (Deep) CFR to remove action bias, then DPO-fine-tune the LLM toward the learned policy and regenerate/expand the latent space in each iteration.
Key results
- LSPO agent win rate as Werewolf 0.73 and as Village 0.27 (iteration 3) vs. best baseline SLA at 0.69 and 0.25.
- Overall win rate of LSPO agent 0.50 compared with 0.47 for SLA and 0.38 for ReAct.
- In Rock-Paper-Scissors-Spock-Lizard, LSPO iteration 3 reaches the Nash equilibrium while ReAct and SLA do not.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| iterate LSPO from iteration 1 to iteration 3 | LSPO · Werewolf, Werewolf side | win rate: 0.54 \pm 0.13 -> 0.73 \pm 0.11 | Helped |
| iterate LSPO from iteration 1 to iteration 3 | LSPO · Werewolf, Village side | win rate: 0.18 \pm 0.09 -> 0.27 \pm 0.11 | Helped |
| use LSPO instead of SLA | Werewolf agents · Werewolf, overall | win rate: 0.47 \pm 0.10 -> 0.50 \pm 0.11 | Helped |
| add DPO fine-tuning (ablation) | LSPO · Werewolf, Iter. 1, Village side | win rate: 0.15 \pm 0.09 -> 0.18 \pm 0.09 | Helped |
| add DPO fine-tuning (ablation) | LSPO · Werewolf, Iter. 1, Werewolf side | win rate: 0.47 \pm 0.14 -> 0.54 \pm 0.13 | Helped |
| add latent-space policy optimization (ablation) | LSPO · Werewolf, Iter. 1, Village side | win rate: 0.12 \pm 0.07 -> 0.18 \pm 0.09 | Helped |
| add latent-space policy optimization (ablation) | LSPO · Werewolf, Iter. 1, Werewolf side | win rate: 0.38 \pm 0.16 -> 0.54 \pm 0.13 | Helped |
| increase initial cluster count from k=1 to k=3 (ablation) | LSPO · Four-player Werewolf, Iter. 1 | Werewolf win rate: 0.13 \pm 0.06 -> 0.23 \pm 0.09 | Helped |
| vary initial cluster count from k=1 to k=3 (ablation) | LSPO · Four-player Werewolf, Iter. 3 | Werewolf win rate: 0.24 \pm 0.11; 0.25 \pm 0.08; 0.25 \pm 0.07 | No clear effect |
| vary DPO beta among the tested settings (ablation) | LSPO · Four-player Werewolf | Werewolf win rate | No clear effect |
Surprising findings
- Win-rate gains between iterations are modest and within large standard deviations (e.g., 0.73 ± 0.11 vs. 0.69 ± 0.12 for SLA), so differences from baselines are not clearly significant.
- Performance converges after about five iterations, far below the theoretical bound of the full language action space.
Limitations
- Evaluation uses fixed-opponent head-to-head games with only 100 games per setting and large variance.
- Relies on prompting and clustering; latent strategies depend on k-means over sentence embeddings and a fixed number of clusters per iteration.
- Only one base model (Llama-3-8B-Instruct) and a text-only, seven-player setting were tested.
Experimental setup
Datasets: self-play Werewolf game trajectories
Benchmarks: Werewolf (7-player) Rock-Paper-Scissors-Spock-Lizard
Metrics: win rate prediction accuracy exploitability action distribution
Baselines: ReAct ReCon Cicero-like agent Strategic Language Agent (SLA) w/o Fine-Tuning w/o Policy Learning
Builds on: Counterfactual Regret Minimization Deep CFR Direct Preference Optimization ReAct Strategic Language Agent
Closest papers by meaning
- Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game (ICML 2024 · similarity 0.94)
- Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game (NeurIPS 2025 · similarity 0.92)
- Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolf (NeurIPS 2024 · similarity 0.90)
- Reinforcing LLM Agents via Policy Optimization with Action Decomposition (NeurIPS 2024 · similarity 0.87)
- Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.86)
- MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.