Research map / Large language models / Reasoning and chain of thought

Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Process reward models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an entropy-uncertainty-driven pipeline for generating and annotating process reward data (UnPRM40K) and two uncertainty-aware output aggregation methods (HMR and WRF vote) that combine majority vote with PRM scores for math reasoning.

Problem

Constructing high-quality step-level process reward data for PRMs is expensive (human annotation) or inefficient (Monte Carlo/LLM-judge annotation), and majority vote and PRM-based answer selection each have failure modes in math reasoning.

Approach

Uses token-level entropy of sampled solutions to pick the most uncertain correct and incorrect solutions for data generation, locates the most uncertain error step via uncertainty deltas plus an MC-perplexity adaptive sampling check, and aggregates answers with hybrid (majority-then-PRM) or weighted frequency-plus-PRM-reward voting.

Key results

• On ProcessBench, Qwen2.5-Math-7B-UnPRM40K reaches 40.3 average F1 versus 36.7 for SimPRM40K and 23.5 for RanPRM40K, using 40K examples.
• UnPRM40K's uncertainty-driven annotation uses 52% fewer verified steps, 34% fewer sampled instances and 40% fewer generated tokens than adaptive binary search on the same 1,500 solutions.
• WRF and HMR vote consistently outperform standard majority vote and standard PRM vote across PRMs on MATH and GSMPlus with Qwen2.5-Math-7B-Instruct reasoner at 128 samples.

Surprising findings

• Standard PRM selection can underperform majority vote, and the uncertainty-aware aggregators give substantial gains exactly in those cases.
• Uncertainty-driven annotation achieves ProcessBench performance slightly below EpicPRM40K (first-error annotation) despite lower cost.

Limitations

• Aggregation gains depend on majority vote being reasonably strong; when majority vote underperforms PRMs (some GSMPlus settings), the hybrid methods give no further improvement.
• Uncertainty-driven annotation is slightly below first-error annotation (EpicPRM40K) on ProcessBench.
• Evaluated mainly on math with a small set of reasoners and PRMs.

Experimental setup

Datasets: MATH GSM8K GSMPlus ProcessBench

Benchmarks: ProcessBench MATH GSMPlus

Metrics: F1 score best-of-N accuracy accuracy

Baselines: Math-Shepherd-PRM-7B RLHFlow-PRM-Mistral-8B RLHFlow-PRM-Deepseek-8B EurusPRM-Stage2-7B Qwen2.5-Math-PRM-7B Qwen2.5-Math-7B-PRM800K EpicPRM40K SimPRM40K RanPRM40K Majority Vote

Builds on: EpicPRM Math-Shepherd self-consistency LoRA

Closest papers by meaning

• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.91)
• Free Process Rewards without Process Labels (ICML 2025 · similarity 0.89)
• VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data (ICML 2025 · similarity 0.89)
• Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · similarity 0.89)
• AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence (ICML 2025 · similarity 0.89)
• Process Reward Model with Q-value Rankings (ICLR 2025 · similarity 0.88)