Research map / Large language models / Reasoning and chain of thought

GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Process reward models · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GenPRM, a generative process reward model that produces CoT plus code verification before judging each step, trained via SFT on 23K MATH-derived rationales with Relative Progress Estimation labels, and scales test-time compute via majority voting.

Problem

Classification-based process reward models give scalar step scores with limited process supervision and generalization, and they cannot leverage LLM generative reasoning or scale test-time compute for verification.

Approach

Reframe PRM step verification as a generative task: the PRM writes CoT analysis and executable Python verification code, receives execution feedback, and outputs a Yes-probability reward; labels come from Relative Progress Estimation (ratio of MC scores of consecutive steps with threshold 0.8) instead of hard MC labels, and multiple sampled verification paths are averaged at inference.

Key results

• GenPRM-7B (Maj@8) reaches 80.5 average F1 on ProcessBench, above Qwen2.5-Math-PRM-72B (78.3) and all PRMs under 72B.
• GenPRM-1.5B (Pass@1) averages 57.3 F1 on ProcessBench vs 36.4 for Skywork-PRM-1.5B; GenPRM-1.5B with Maj@8 reaches 63.4.
• In critique refinement with Gemma-3-12b-it generator, GenPRM-7B raises average accuracy from 48.9 (zero-shot) to 55.4 after three turns, versus 47.2 for DeepSeek-R1-Distill-7B.

Surprising findings

• Direct generative PRM trained on the same data without CoT or code reaches only 60.0 average F1, far below the CoT-based GenPRM at 75.2 (Pass@1).
• Most of the gain comes from CoT reasoning; code verification adds further improvement but CoT-only inference already accounts for much of it.

Limitations

• Generative reasoning adds substantial inference-time computation per step and response.
• Evaluated mainly on mathematical reasoning; coding and general domains are not explored.
• Does not use reinforcement learning to incentivize generative verification reasoning.

Experimental setup

Datasets: MATH ProcessBench AMC23 AIME24 Minerva Math

Benchmarks: ProcessBench MATH AMC23 AIME24 Minerva Math

Metrics: F1 Pass@1 Maj@8 accuracy after critique refinement

Baselines: Math-Shepherd-PRM-7B Skywork-PRM-7B Qwen2.5-Math-PRM-7B Qwen2.5-Math-PRM-72B Universal-PRM-7B Dyve-14B Direct Generative PRM-7B Self-Refine

Builds on: Math-Shepherd GRPO ProcessBench DeepSeek-R1-Distill

Closest papers by meaning

• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.93)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.91)
• StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models (ICLR 2026 · similarity 0.91)
• Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.90)
• VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data (ICML 2025 · similarity 0.90)
• ContextPRM: Leveraging Contextual Coherence for multi-domain Test-Time Scaling (ICLR 2026 · similarity 0.89)