Research map / Large language models / Reasoning and chain of thought
Process Reward Model with Q-value Rankings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PQM, which frames process reward modeling as Q-value ranking under a deterministic MDP and trains PRMs with a margin-based comparative loss, outperforming BCE/MSE PRMs for best-of-n verification on MATH500 and GSM-Plus.
Problem
Existing process reward models are trained as per-step classifiers with cross-entropy loss, treating steps independently and lacking theoretical grounding for the reward they approximate.
Approach
Define a Q-value as the log-odds of reaching a correct final answer from a step, derive an optimal Q-value ranking across correct and wrong steps in a deterministic MDP, and train PRMs with a Plackett-Luce-style comparative loss with a margin zeta between correct and wrong steps.
Key results
- With Llama-3-70B-Instruct samples on MATH500 at BON@128, PQM reaches 51.4% versus 39.8% for BCE (an 11.6-point gain).
- On MetaMath-Mistral-7B samples on GSM-Plus at BON@128, PQM (zeta=2) reaches 66.00% versus 61.72% for BCE.
- PRM-guided beam search on MATH500 with Llama-3.1-8B-Instruct improves Pass@1 from 26.4 (BCE) to 31.6 (PQM).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace BCE with PQM Q-value ranking | Deepseek-7B-base PRM verifying Llama-3-70B-Instruct samples · MATH500 BON@128 | BON@128 accuracy: 39.8 -> 51.4 | Helped |
| replace BCE with PQM Q-value ranking | Deepseek-7B-base PRM · GSM-Plus BON@128 across three sampling policies | BON@128 accuracy | Helped |
| use an appropriate intermediate margin (ζ=2, 4, or 8) rather than extreme ζ=0 or ζ=16 (ablation) | PQM · MATH500 BON@n across three sampling policies | BON@n accuracy | Helped |
| increase margin from ζ=2 to ζ=4 (ablation) | PQM · MATH500 BON@128, MetaMath-Mistral-7B policy | BON@128 accuracy: 43.8 -> 44.6 | Helped |
| increase margin from ζ=2 to ζ=4 (ablation) | PQM · MATH500 BON@128, Llama-3-70B-Instruct policy | BON@128 accuracy: 50.4 -> 48.2 | Hurt |
| increase margin from ζ=4 to ζ=16 (ablation) | PQM · MATH500 BON@128, MetaMath-Mistral-7B policy | BON@128 accuracy: 44.6 -> 39.2 | Hurt |
| replace practical comparative loss with theoretical loss that ranks all negative steps (ablation) | PQM · MATH500 BON@n across three sampling policies | BON@n accuracy | Hurt |
| rank only the first incorrect step instead of all negative steps (ablation) | PQM · MATH500 BON@n | BON@n accuracy | No clear effect |
| combine PQM with self-consistency (ablation) | PQM · MATH500 verification across three policy models | verification accuracy | Helped |
| use a math-specialized Qwen2-Math-1.5B-Instruct backbone instead of the other evaluated backbones (ablation) | PQM · MATH500 BON@n | BON@n accuracy | Mixed |
| train with PQM instead of BCE at reduced training-data fractions (ablation) | Deepseek-7B-base PRM · MATH500 BON@n at 25%, 50%, and 75% training data | BON@n accuracy | Mixed |
| use PQM instead of BCE to guide beam search | Deepseek-7B-base PRM · MATH500 beam search with Llama-3-8B-Instruct and Eurus-7b-sft | beam-search accuracy: 26.4 -> 31.6; 24.2 -> 29.2 | Helped |
Surprising findings
- BCE-trained PRMs produce monotonically decreasing probabilities for consecutive correct steps, contradicting the theorem's predicted ascending Q-values.
- The theoretically derived loss L_theorem, which also ranks wrong steps among themselves, performs worse than the simplified loss that drops that ranking.
Limitations
- Math-Shepherd labels are noisy from automatic annotation, and all steps after the first error are marked wrong.
- Gap remains between PRM verification and the pass@N ceiling.
- Online RL with PRMs is not explored.
Experimental setup
Datasets: Math-Shepherd Math-Step-DPO-10K
Benchmarks: MATH500 GSM-Plus MATH500 beam search
Metrics: BON@n accuracy Pass@1
Baselines: ORM BCE (Math-Shepherd) MSE with 0-1 labels MSE with MCTS labels
Builds on: Math-Shepherd Plackett-Luce ranking model DPO self-consistency
Closest papers by meaning
- Free Process Rewards without Process Labels (ICML 2025 · similarity 0.90)
- Discriminative Policy Optimization for Token-Level Reward Models (ICML 2025 · similarity 0.89)
- Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning (AAAI 2026 · similarity 0.88)
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.88)
- Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.87)
- Know What You Don't Know: Uncertainty Calibration of Process Reward Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.