Research map / Multi-agent systems and games / Games and equilibria
SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
Bo Liu, Simon Yu, Zichen Liu, Leon Guertler, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, Natasha Jaques
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-game SPIRAL self-play training | Qwen3-4B-Base · Reasoning suite average (8 benchmarks) | average accuracy: 34.0 -> 44.5 | Improved |
| add multi-game SPIRAL self-play training | Qwen3-8B-Base · Reasoning suite average (8 benchmarks) | average accuracy: 39.5 -> 49.6 | Improved |
| add SPIRAL to an already instruction-tuned model | Llama-3.1-8B-Instruct · Reasoning suite average (8 benchmarks) | average accuracy: 23.9 -> 25.9 | Improved |
| add SPIRAL to a model already RLVR-trained | DeepSeek-Distill-Qwen-7B · Reasoning suite average (8 benchmarks) | average accuracy: 60.4 -> 61.8 | Improved |
| replace SFT on expert game trajectories with SPIRAL self-play (ablation) | Qwen3-4B-Base · Reasoning suite average (8 benchmarks) | average accuracy: 39.7 -> 44.5 | Improved |
| scale SFT data from 25k to 52k trajectories (ablation) | SFT-Multi on Qwen3-4B-Base · Reasoning suite average (8 benchmarks) | average accuracy: 39.7 -> 39.7 | No clear effect |
| SFT hurts when applied to an RLVR-trained model | DeepSeek-Distill-Qwen-7B · Reasoning suite average (8 benchmarks) | average accuracy: 60.4 -> 58.3 | Hurt |
| use self-play instead of fixed Mistral opponent (ablation) | SPIRAL on Kuhn Poker · Reasoning suite average (8 benchmarks) | average accuracy: 29.6 -> 43.4 | Improved |
| use self-play instead of fixed Gemini opponent (ablation) | SPIRAL on Kuhn Poker · Reasoning suite average (8 benchmarks) | average accuracy: 33.4 -> 43.4 | Improved |
| train on three games instead of one (add games) (ablation) | SPIRAL on Qwen3-4B-Base · Reasoning suite average (8 benchmarks) | average accuracy: 40.0 -> 44.5 | Improved |
| add Simple Negotiation to TicTacToe+KuhnPoker training (ablation) | SPIRAL on Qwen3-4B-Base · Reasoning suite average (8 benchmarks) | average accuracy: 41.3 -> 44.5 | Improved |
| add multi-game training over single-game specialists (ablation) | SPIRAL game-play · Game win rate vs Gemini-2.0-Flash (average over 6 games) | win rate (%): 52.9% -> 59.5% | Improved |
Closest papers by meaning
- MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs (ICLR 2026 · similarity 0.92)
- Absolute Zero: Reinforced Self-play Reasoning with Zero Data (NeurIPS 2025 · similarity 0.89)
- SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning (NeurIPS 2025 · similarity 0.89)
- Play to Generalize: Learning to Reason Through Game Play (ICLR 2026 · similarity 0.89)
- Sparks of Cooperative Reasoning: LLMs as Strategic Hanabi Agents (ICML 2026 · similarity 0.88)
- Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
