Research map / Large language models / Reasoning and chain of thought

Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Process reward models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Conditional Reward Modeling (CRM), which models step rewards as conditional probabilities of entering a wrong state, linked to the final outcome via the chain rule and PBRS, and shows gains in Best-of-N, beam search, and RL with less reward hacking.

Problem

Existing process reward models either score steps in isolation without modeling inter-step dependencies, or fail to tie step rewards to the final outcome, leading to ambiguous credit assignment and reward hacking in RL.

Approach

Frame reasoning as a temporal process where the first wrong-state step z follows a distribution; define step reward as the conditional hazard h(t)=Pr(z=t|z>=t), so that the log(1-h(t)) rewards telescope to the trajectory success probability S(T), with PBRS using potential log S(t).

Key results

• On MATH500 with Qwen2.5-3B-Instruct, CRM reaches 56.6% at Best-of-32, ahead of the strongest baseline by 1.4 points.
• In VR-disabled RL, CRM reaches 43.3% Pass@1 on AIME24, exceeding PURE by 16.7 points.
• CRM outperforms PRM, PQM and IPRM in beam search on MATH500 and Gaokao2023 across most settings, with the gap widening as N grows to 100.

Surprising findings

• PRM and PQM show reward rising while downstream accuracy declines, driven by long repetitive outputs during RL.
• Log-transformed PRM/PQM baselines and a linearized CRM variant also collapse into reward hacking, indicating the log(1-h) form matters.

Limitations

• Relies on step-level process labels (Math-Shepherd) for training the reward model.
• Evaluation mostly on math reasoning with limited multi-domain validation.
• Some Best-of-N results are mixed; PQM is best on some GSM-Plus settings.

Experimental setup

Datasets: Math-Shepherd Orz-Math-57k MMLU-Pro-CoT-Train GSM-Plus MATH500 Gaokao2023 MMLU-Pro-CoT-Eval

Benchmarks: GSM-Plus MATH500 Gaokao2023 AIME24 AIME25 AMC23 Minerva Math OlympiadBench MMLU-Pro-CoT-Eval

Metrics: accuracy Best-of-N accuracy Pass@1 AUPRC repeat score self-reflection score

Baselines: ORM PRM PQM IPRM PURE Prime VR R-PRM GenPRM

Builds on: Math-Shepherd PQM IPRM Potential-Based Reward Shaping RLOO

Closest papers by meaning

• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.94)
• Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.92)
• Know What You Don't Know: Uncertainty Calibration of Process Reward Models (NeurIPS 2025 · similarity 0.92)
• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.91)
• ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs (NeurIPS 2025 · similarity 0.91)
• Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · similarity 0.90)