Research map / Large language models / Reasoning and chain of thought

Adversarial Training for Process Reward Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Process reward models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes APRM, a two-player general-sum game in which a PRM and an adversarial step-corrupting generator are co-trained with PPO plus OGDA, improving math solver accuracy over PRM and prompting baselines.

Problem

PRMs need step-level supervision, but manual step labels are expensive and static training data generalizes poorly to novel reasoning errors.

Approach

Formulates PRM training as a general-sum game where a generator learns to corrupt correct steps so they fool the PRM, while the PRM learns to detect them; regularized utilities give a stationary-point Nash equilibrium with linear convergence under OGDA.

Key results

• Averaged over five math benchmarks and four solver models, APRM improves solver accuracy by +3.4 pp over the strongest trained PRM baseline (ReST-MCTS).
• On JEEBench (OOD), APRM outperforms ReST-MCTS by +5.3 pp.
• RL post-training of Gemma-3-12B with APRM reward reaches 86.8 on MATH500 vs 80.0 for outcome reward.

Surprising findings

• ReST-MCTS-trained PRM gains shrink and become negative with larger solver (-7.6 pp on Gemma-3-27B vs prompting), suggesting static-data PRMs fail to scale with solver strength.
• The paper's own failure analysis shows APRM misses errors violating theorem preconditions, such as misapplied Alternating Series Test and Jensen's inequality.

Limitations

• Training requires additional compute for adversarial optimization, higher than static-dataset approaches.
• Generator struggles to produce adversarial negatives that violate theorem preconditions, possibly due to backbone size.
• Convergence theorem is in policy space; the paper does not claim gradient training finds the equilibrium, and the convergence proof is sketchy.

Experimental setup

Datasets: MATH train split PRM800K test subset

Benchmarks: MATH500 JEEBench OlympiadBench AIME25 AMC SciBench PRM800K

Metrics: accuracy precision recall F1

Baselines: CoT-SC LLM-as-a-Judge Tree of Thoughts ReST-MCTS Hard Negatives ARM AutoPSV Let's Verify Step by Step Math-Shepherd rStar-Math

Builds on: PPO Optimistic Gradient Descent-Ascent Generative Adversarial Networks Math-Shepherd

Closest papers by meaning

• Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning (ICLR 2026 · similarity 0.89)
• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.89)
• Free Process Rewards without Process Labels (ICML 2025 · similarity 0.89)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.89)
• AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence (ICML 2025 · similarity 0.88)
• Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs (NeurIPS 2025 · similarity 0.87)