Research map / Large language models / Reasoning and chain of thought
SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Play Critic (SPC), where a sneaky step generator and a step-level critic (both fine-tuned from Qwen2.5-7B-Instruct) play adversarial games whose outcomes give RL rewards, improving step-correctness critique and LLM test-time search without human step labels.
Problem
Evaluating step-level reliability of LLM chain-of-thought reasoning requires costly human step-level annotation, which becomes outdated as LLMs change and limits critics to scoring rather than substantive critique.
Approach
Two copies of a base model play an adversarial game: a sneaky generator converts correct steps into subtle errors that also measurably reduce solver success, and a critic must detect them. Win/loss outcomes provide rewards for offline RL of both roles, and the critic is also used to guide step-by-step test-time search with regeneration on errors.
Key results
- On ProcessBench, SPC average accuracy rises from 70.8% (round 0) to 77.7% (round 2), above DeepSeek-R1-Distill-Qwen-7B at 75.2%.
- On PRM800K, SPC round 2 averages 75.8% versus 71.4% for distilled R1-7B.
- With Self-Consistency, SPC lifts Qwen2.5-32B-Instruct on AIME2024 to 23.3%, versus 16.7% for Self-Consistency with Qwen2.5-Math-7B-PRM800K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Iteratively train the critic through adversarial self-play from Round 0 to Round 2 | SPC critic · ProcessBench | Average accuracy: 70.8% to 77.7% | Helped |
| Iteratively train the critic through adversarial self-play from Round 0 to Round 2 | SPC critic · PRM800K | Average recall: 71.0% to 75.8% | Helped |
| Iteratively train the critic through adversarial self-play from Round 0 to Round 2 | SPC critic · DeltaBench | Average recall: 54.9% to 60.5% | Helped |
| Guide test-time search with SPC and Self-Consistency | Qwen2.5-32B-Instruct · AIME2024 | Accuracy: 16.7 to 23.3 | Helped |
| Guide test-time search with SPC and Self-Consistency | DeepSeek-R1-Distill-Qwen-7B · MATH500 | Accuracy: 91.8 to 94.0 | Helped |
| Construct paired positive and negative critiques for each successfully transformed erroneous step (ablation) | SPC critic · ProcessBench | Average accuracy: 75.0% to 76.8% | Helped |
| Generate round 2 data against Critic-1 rather than Critic-0 (ablation) | SPC critic · ProcessBench | Average accuracy: 76.8% to 72.0% | Hurt |
| Generate round 2 data against Critic-0 rather than Critic-1 (ablation) | SPC critic · ProcessBench | Average accuracy: 77.7% | Helped |
| Remove failed attacks on the solver as negative samples when training the sneaky generator (ablation) | SPC sneaky generator · Qwen2.5-7B-Instruct solver | Successful attack rate: 12.1% | Hurt |
| Train the sneaky generator without solver-failure negatives (ablation) | SPC sneaky generator · Critic-0 | Win rate: 19.6% | Hurt |
Surprising findings
- Playing the sneaky generator against an equally matched critic (round 1 vs. round 0) worked better than synchronous self-play; unbalanced games degraded performance from 76.8% to 72.0%.
- Removing solver-based rewards for the sneaky generator sharply reduced successful attacks (to 12.1%), showing the solver signal is essential.
Limitations
- Experiments are limited to mathematical reasoning tasks.
- Sneaky-step validity depends on repeated solver rollouts with a strict success-rate criterion, which is costly.
- Training relies on GPT-4 and GPT-4o generated data and DeepSeek-R1-Distill-Qwen-7B critiques.
Experimental setup
Datasets: PRM800K GSM8K MATH OlympiadBench Omni-MATH
Benchmarks: ProcessBench PRM800K DeltaBench MATH500 AIME2024
Metrics: recall accuracy average recall harmonic mean problem-solving accuracy
Baselines: Math-Shepherd-PRM-7B Qwen2.5-Math-7B-PRM800K Llama-3.1-8B-Instruct Llama-3.1-70B-Instruct Qwen2.5-7B-Instruct Qwen2.5-32B-Instruct GPT-4o DeepSeek-R1-Distill-Qwen-7B Self-Consistency
Builds on: RLOO GRPO Math-Shepherd PRM800K ProcessBench
Closest papers by meaning
- Self-playing Adversarial Language Game Enhances LLM Reasoning (NeurIPS 2024 · similarity 0.91)
- Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning (NeurIPS 2026 · similarity 0.90)
- Improving Rationality in the Reasoning Process of Language Models through Self-playing Game (ICML 2025 · similarity 0.90)
- Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.90)
- Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning (ICLR 2026 · similarity 0.89)
- SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.