Research map / Large language models / Language, safety and interpretability
Self-playing Adversarial Language Game Enhances LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPAG, where LLMs play the adversarial Taboo word game against a copy of themselves, then are trained via imitation of GPT-4 games and offline RL on winning episodes, improving reasoning benchmarks over three epochs.
Problem
Improving the general reasoning ability of LLMs without relying on curated reasoning datasets or subjective self-judgments, which are costly and can amplify model biases.
Approach
Model the Adversarial Taboo game as a zero-sum two-player Markov game where an LLM plays both attacker and defender via self-play, with game outcomes (automatically checkable) providing rewards; winners' episodes are used for offline RL (ReST-style) plus an SFT regularizer.
Key results
- On LLaMA-2-7B, SPAG-3 raises the reasoning geometric mean from 49.17 (base) to 52.58, beating AlpacaSFT-3 (50.08) and CoT (50.05).
- On Baichuan-2-13B, SPAG-3 reaches a GM of 56.75 versus 54.21 for the base model and 51.79 for AlpacaSFT-3.
- SPAG-trained models win more often against GPT-4 as the game player, and defender win rate against the previous-epoch checkpoints rises across epochs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with SPAG self-play through epoch 3 rather than use IM-AlpacaSFT | LLaMA-2-7B · Reasoning benchmarks, GM (Avg.) | geometric mean: 51.20 -> 52.58 | Helped |
| Train with SPAG self-play through epoch 3 rather than use IM+AlpacaSFT | Baichuan-2-13B · Reasoning benchmarks, GM (Avg.) | geometric mean: 55.80 -> 56.75 | Helped |
| Increase SPAG training from epoch 1 to epoch 3 | LLaMA-2-7B · Reasoning benchmarks, GM (Avg.) | geometric mean: 51.69 -> 52.58 | Helped |
| Increase SPAG training from epoch 1 to epoch 3 | Baichuan-2-13B · Reasoning benchmarks, GM (Avg.) | geometric mean: 56.09 -> 56.75 | Helped |
| Use SPAG self-play instead of continuous SFT | LLaMA-2-7B IM model · Reasoning benchmarks, GM (Avg.) | geometric mean: 51.20 vs 50.22 | Helped |
| Use SPAG self-play instead of continuous SFT | Baichuan-2-13B IM model · Reasoning benchmarks, GM (Avg.) | geometric mean: 56.09 vs 55.80 | Helped |
| Increase SPAG training from epoch 1 to epoch 3 | Baichuan-2-13B · MMLU | accuracy: 57.93 -> 57.75 | Hurt |
| Increase SPAG training from epoch 1 to epoch 3 | LLaMA-2-7B · WinoGrande | accuracy: 68.90 -> 69.93 | Helped |
| Use SPAG-3 instead of AlpacaSFT-3 (ablation) | LLaMA-2-7B · Reasoning benchmarks | benchmark scores | Helped |
| Increase imitation data beyond 5K episodes (ablation) | GPT-4 imitation learning · Reasoning benchmarks | reasoning performance | No clear effect |
| Set imitation KL coefficient around 0.1 (ablation) | GPT-4 imitation learning · Reasoning benchmarks | reasoning performance | Helped |
| Increase SPAG SFT coefficient above 0.5 (ablation) | SPAG · Reasoning benchmarks | reasoning performance | No clear effect |
Surprising findings
- MMLU does not improve consistently with SPAG; Baichuan-2-13B's MMLU slightly decays during self-play training.
- Non-adversarial self-play on Guess-My-City (SP-GuessCity) performs worse than its imitation model on reasoning benchmarks.
Limitations
- Only verified on LLaMA-2-7B and Baichuan-2-13B; larger models not tested.
- Value function and advantage estimation are approximated; more accurate estimators (MCTS, actor-critic) unexplored.
- Potential for harmful adversarial behaviors such as cheating or bluffing has not been studied.
Experimental setup
Datasets: CoCA top-50K word frequency list Alpaca 52K instruction data GPT-4 self-play Adversarial Taboo episodes 20-Question keyword list (168 test words) BBH ARC Mutual WinoGrande LogiQA2 PIQA MMLU
Benchmarks: MMLU BBH Mutual ARC-e ARC-c LogiQA2 WinoGrande PIQA
Metrics: accuracy geometric mean across benchmarks game win rate
Baselines: Chain-of-Thought prompting AlpacaSFT Imitation-20Q Imitation-GuessCity SP-20Q SP-GuessCity IM+AlpacaSFT GPT-4
Builds on: ReST AlphaGo Zero Adversarial Taboo A-LoL PPO
Closest papers by meaning
- SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
- Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning (NeurIPS 2026 · similarity 0.88)
- Improving Rationality in the Reasoning Process of Language Models through Self-playing Game (ICML 2025 · similarity 0.87)
- SeRL: Self-play Reinforcement Learning for Large Language Models with Limited Data (NeurIPS 2025 · similarity 0.87)
- GameArena: Evaluating LLM Reasoning through Live Computer Games (ICLR 2025 · similarity 0.87)
- MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.