Research map / Large language models / Reasoning and chain of thought
Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Critic-Discernment Game where a prover is challenged by helpful or misleading critiques and trained via ReST self-play to improve reasoning rationality without human or superior-model supervision.
Problem
LLMs lack true comprehension of their own reasoning processes, producing errors and hallucinations they cannot detect or correct, and existing step-level supervision (PRMs, step-wise preferences) relies on human or superior-model annotations.
Approach
A three-role self-play game (prover, helpful critic, misleading critic) where rewards come from game rules: the prover must keep correct answers under misleading critiques and revise incorrect ones under helpful critiques; all roles are trained with ReST-style rejection-sampled fine-tuning over offline rounds.
Key results
- On MATH500, Llama-3.1-8B-Instruct improves from 49.4 to 51.7 Pass@1 and 63.4 to 66.0 M@32 after two CDG rounds.
- Stepwise error detection F1 on MATH500 rises from 64.4 to 71.4 for Llama-3.1-8B-Instruct with CDG.
- Overall average across four tasks rises from 42.5 (vanilla) to 45.5 (CDG), above Expert Iteration (43.4) and Step-DPO (41.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train for two rounds with CDG self-play | Llama-3.1-8B-Instruct · GSM8K | P@1: 85.3 -> 86.8 | Helped |
| Train for two rounds with CDG self-play | Llama-3.1-8B-Instruct · MATH500 | P@1: 49.4 -> 51.7 | Helped |
| Train for one round with CDG self-play | Qwen2.5-1.5B-Instruct · MATH500 | P@1: 55.4 -> 57.6 | Helped |
| Train the base model with CDG self-play for two rounds after imitation learning | Llama-3.1-8B-Base · MATH500 | P@1: 29.4 -> 33.8 | Helped |
| Apply CDG training | Llama-3.1-8B-Instruct · MATH500 stepwise error detection | accuracy: 55.4 -> 67.5 | Helped |
| Apply CDG training | Llama-3.1-8B-Instruct · GSM8K stepwise error detection | F1: 74.0 -> 76.9 | Helped |
| Add the helpful critic to CDG training (ablation) | CDG prover · GSM8K math reasoning | P@1: 86.2 -> 86.8 | Helped |
| Add the misleading critic to CDG training (ablation) | CDG prover · GSM8K math reasoning | P@1: 84.9 -> 86.8 | Helped |
| Add both critic types rather than use only one critic (ablation) | CDG prover · MATH self-correction | self-correction: -3.0 -> 1.4 | Helped |
| Use DPO instead of ReST for CDG training (ablation) | CDG prover · MATH500 | P@1: 51.7 -> 46.0 | Hurt |
| Use PPO instead of ReST for CDG training (ablation) | CDG prover · MATH500 | M@32: 66.0 -> 62.6 | Hurt |
| Train with Expert Iteration instead of CDG | Llama-3.1-8B-Instruct · GSM8K math reasoning | P@1: 85.3 -> 87.2 | Helped |
Surprising findings
- A single CDG round gives limited or no math reasoning improvement on Llama-3.1-8B-Instruct; gains appear mainly in the second round.
- Single-critic variants (helpful-only or misleading-only) can degrade self-correction below the vanilla model.
Limitations
- Gains on math reasoning are modest (e.g., about 1.7 points Pass@1 on MATH500) and DPO variant underperforms the vanilla model.
- DPO training is highly sensitive to hyperparameters and often produces repetitive text.
- Evaluation is limited to mathematical tasks and a small set of models.
Experimental setup
Datasets: GSM8K MATH MATH500
Benchmarks: GSM8K MATH500
Metrics: Pass@1 maj@8 maj@32 F1 accuracy self-correction rate win rate
Baselines: Expert Iteration Step-DPO CDG-Imitation CDG w/o Correct critic CDG w/o Misleading critic CDG-DPO CDG-PPO
Builds on: ReST Expert Iteration SPAG DPO PPO
Closest papers by meaning
- SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.89)
- Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability (ICML 2026 · similarity 0.89)
- Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.89)
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.89)
- On the self-verification limitations of large language models on reasoning and planning tasks (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.