Research map / Multi-agent systems and games / Games and equilibria
Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-agent KTO (MaKTO), training LLMs for the Werewolf social deduction game via behavior cloning on expert annotated data followed by KTO on stepwise-selected preference data from multi-agent gameplay.
Problem
Language agents in social deduction games like Werewolf typically separate language generation from strategic decision-making, and win/loss outcomes are too coarse to judge individual actions; paired preference data is hard to obtain in multi-agent dialogue.
Approach
Use KTO (unpaired preference learning) on stepwise desirable/unacceptable actions and speeches, selected via heuristic rules, staged voting outcomes, and GPT-4o verification, collected from diverse multi-agent gameplay rather than self-play, after SFT on expert chain-of-thought data.
Key results
- MaKTO-72b achieved a 61% average win rate across head-to-head Seer-Witch-Guard games, outperforming GPT-4o and the two-stage Mix agent by relative improvements of 23.0% and 10.9%.
- MaKTO-72b won 60% of 20 head-to-head games against 14 expert human players (in a limited-sample setting).
- Human detection accuracy of MaKTO-72b was 48.9%, versus 76.6% for GPT-4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MaKTO training to Qwen2.5-72b-instruct | Qwen2.5-72b-instruct · 9-player Seer-Witch-Guard | average win rate: 0.496 -> 0.610 | Helped |
| apply MaKTO training to Qwen2.5-72b-instruct | Qwen2.5-72b-instruct · Human-AI head-to-head, 9-player Seer-Witch-Guard | win rate: 60% | Helped |
| apply MaKTO training to Qwen2.5-72b-instruct | Qwen2.5-72b-instruct · Turing-style detectability test | detection accuracy: 76.6% -> 48.9% | Helped |
| apply MaKTO training to Qwen2.5-72b-instruct | Qwen2.5-72b-instruct · Human-annotated test set | voting accuracy: 66.5% -> 73.8% | Helped |
| apply MaKTO training to Qwen2.5-72b-instruct | Qwen2.5-72b-instruct · 9-player Seer-Witch-Hunter | average win rate: 0.507 -> 0.563 | Helped |
| replace multi-agent gameplay with self-play (ablation) | 14B MaKTO · 9-player Seer-Witch-Guard | average win rate: 0.534 -> 0.483 | Hurt |
| replace actual gameplay with annotated data only (ablation) | 14B MaKTO · 9-player Seer-Witch-Guard | average win rate: 0.534 -> 0.374 | Hurt |
| select preference data by trajectory outcomes instead of stepwise actions (ablation) | 14B MaKTO · 9-player Seer-Witch-Guard | average win rate: 0.545 -> 0.458 | Hurt |
Surprising findings
- Preference data built only from annotations without actual gameplay reduced average win rate by 13%.
- Trajectory-level win/loss selection performed worse than stepwise action selection for KTO training.
Limitations
- Relies on turn-based conversation rather than free-form multi-agent communication.
- Hallucinations and inconsistent behavior over long conversations.
- Offline method; online multi-agent RL not explored.
Experimental setup
Datasets: expert-annotated Werewolf gameplay dataset (331 games) game terminology and strategy guides
Benchmarks: 9-player Seer-Witch-Guard Werewolf tournament 9-player Seer-Witch-Hunter generalization tournament human-AI Werewolf games Turing-style detectability test
Metrics: win rate TrueSkill rating voting accuracy abstention rate side alignment accuracy wolf-prediction F1 human detection accuracy
Baselines: GPT-4o GPT-4o-mini Claude-3.5-Sonnet Mix agent SFT-72b Self-play KTO
Builds on: KTO DPO TrueSkill
Closest papers by meaning
- Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game (ICML 2024 · similarity 0.93)
- Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization (ICML 2025 · similarity 0.92)
- Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolf (NeurIPS 2024 · similarity 0.89)
- MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs (ICLR 2026 · similarity 0.88)
- $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.87)
- Self-Challenging Language Model Agents (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.