Research map / Large language models / Reasoning and chain of thought

Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scaling

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Process reward models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives a MAP-based optimal weighted majority vote combining PRM scores and LLM reliability, then proposes calibrated (KDE, logit, linear) weighting functions that learn negative weights for low-PRM responses to improve test-time scaling efficiency.

Problem

Process reward models often fail to beat simple majority voting in test-time scaling because standard best-of-N and voting schemes do not optimally combine PRM scores with LLM answer consensus.

Approach

Shows under conditional independence assumptions that the optimal aggregation is a weighted majority vote whose per-response weight is a PRM likelihood-ratio term plus an LLM reliability term; calibrates a global weighting function (logit or linear with zero-crossing offset b, or KDE) on a small labeled set, allowing negative weights for low-scoring responses.

Key results

• Logit WV matches or beats vanilla weighted voting while using about 37.1% (MATH) and 21.3% (MATH500) of its test-time compute on average across 35 LLM-PRM pairs.
• On Llama3.1-Mistral-8B PRM, Logit WV beats the best baseline Vanilla WV by about 3 points on MATH (61.2 vs 58.2 average across 5 LLMs).
• Calibrated optimal weighting functions consistently assign negative offsets; the optimal zero-crossing b is above zero for all LLMs.

Surprising findings

• Optimal weighting assigns negative weights to low-PRM-score responses, so they count against their answers.
• Per-question optimal weighting functions vary widely and a learned meta-model could not predict them, so global calibration stays suboptimal.

Limitations

• Theory relies on conditional independence assumptions between generations and between scores.
• Calibration learns a single global weighting function, while the optimal weights vary per question.
• Evaluation is limited to mathematical reasoning and requires a small labeled calibration set.

Experimental setup

Datasets: MATH MATH500 MMLU-Pro-CoT-Eval

Benchmarks: MATH MATH500 MMLU-Pro-CoT-Eval

Metrics: accuracy

Baselines: Majority Vote Best-of-N Vanilla Weighted Vote Pass@N CISC

Builds on: self-consistency confidence-informed self-consistency best-of-N

Closest papers by meaning

• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.91)
• Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.89)
• Know What You Don't Know: Uncertainty Calibration of Process Reward Models (NeurIPS 2025 · similarity 0.89)
• T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Models (ICLR 2026 · similarity 0.89)
• Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimality (ICLR 2026 · similarity 0.89)
• Scaling Test-Time Compute Without Verification or RL is Suboptimal (ICML 2025 · similarity 0.88)