Research map / Large language models / Reasoning and chain of thought

Know What You Don't Know: Uncertainty Calibration of Process Reward Models

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Process reward models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows off-the-shelf process reward models overestimate success probabilities, calibrates them via quantile-regression fine-tuning on Monte Carlo success labels, and uses the calibrated lower quantiles to drive instance-adaptive best-of-N and beam search that save compute on math benchmarks.

Problem

Off-the-shelf process reward models (PRMs) are miscalibrated, overestimating the probability that a partial reasoning step leads to a correct final answer, which undermines using them for uncertainty estimates and adaptive compute allocation in inference-time scaling.

Approach

Fine-tune the PRM with a quantile-regression head (weighted quantile loss over 10/50/90% levels) on Monte Carlo success-rate labels so that a lower quantile gives conservative success estimates, then use these to set per-instance sample budgets (instance-adaptive scaling) for best-of-N and beam search with a theoretical sample-complexity bound.

Key results

• Quantile-regression calibration reduces Brier score for Qwen2.5-Math-PRM-7B on MATH500 for Llama-3.2-1B from 0.2414 to 0.0692 (and Qwen-2.5-7B from 0.1008 to 0.0818).
• BoN+IAS with calibrated PRM uses about 0.23-0.64 of the N=64 budget on MATH500 while keeping accuracy close to fixed-N BoN (e.g., Qwen-2.5-7B 0.8368 vs 0.8540 BoN).
• Beam search IAS (IASoM) reduces budget to about 0.36-0.99 of fixed BS while matching or improving accuracy on several MATH500 models (e.g., R1-Qwen-7B 0.8740 vs 0.8280 BS).

Measured effects reported in the paper

Surprising findings

• Uncalibrated PRMs make instance-adaptive BoN degrade accuracy for weaker models, because overestimated success probabilities cause overly aggressive downsampling.

Limitations

• Evaluated only on mathematical reasoning benchmarks (MATH500, AIME24-25).
• Monte Carlo labels use only N_MC=8 rollouts per prefix, which adds label noise to the training target.
• The method only assigns maximum budget to hard problems and does not consider abstaining or routing to stronger models.

Experimental setup

Datasets: MATH training split MATH500 AIME24-25

Benchmarks: MATH500 AIME24-25

Metrics: Brier score positive Brier score AdaptiveCE ECE AverageCE Pass@1 relative budget ratio

Baselines: temperature scaling isotonic regression histogram binning fixed-N best-of-N fixed-budget beam search

Builds on: temperature scaling conformal quantile regression LoRA Qwen2.5-Math-PRM

Closest papers by meaning

• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.92)
• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.92)
• Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.91)
• Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling (ICLR 2026 · similarity 0.90)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.90)
• A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.90)