Research map / Large language models / Reasoning and chain of thought

Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes why inference-time reward models underperform in reasoning (easy questions, many samples, high diversity) and proposes CRISP, a cluster-based, prefix-guided search that aggregates reward by final answer.

Problem

Reward models used for inference-time selection (BoN, MCTS) can hurt LLM reasoning: they degrade on easy questions, their discriminative ability declines as sample count grows, and high sampling diversity further harms them.

Approach

CRISP samples full reasoning continuations conditioned on a prefix set, clusters paths by final answer, sums normalized RM scores per cluster (so frequency counts), terminates early when few clusters remain (reverting to majority vote), and extracts top-path prefixes for the next round.

Key results

• On Llama3.1-8B MATH with PRM, CRISP reaches 0.67 vs 0.62 for the best RM baseline (BoN+PRM), up to 5% gain over RM-based inference methods.
• Versus DeepSeek-R1-Distill on Qwen2.5-Math-7B, CRISP averages about 10% higher accuracy on commonsense tasks with roughly 90% fewer tokens in the best case (e.g., 1100 vs 3612 tokens on CSQA).
• On MATH with Qwen2.5-3B + Skywork-o1 PRM over five runs, CRISP averages 0.768 vs 0.714 for MCTS and 0.652 for BoN.

Surprising findings

• Reward-model-based BoN underperforms plain self-consistency on easy questions, and on Qwen2.5-3B BoN with all reward models scores below SC on MATH.
• Oracle (coverage) keeps rising with sample count while RM-selected accuracy plateaus, indicating the RM, not the generator, is the bottleneck.

Limitations

• Analysis limited to reward models trained on paired data; the paper explicitly leaves reward-model training effects to future work.
• Hyperparameters (threshold, top-k, max steps) are sensitive and task-dependent, and several results show gains of only a few points within run-to-run variance.

Experimental setup

Datasets: MATH-500 GSM8K OlympiadBench CSQA SIQA LogiQA AQuA WinoGrande ProofWriter ProntoQA

Benchmarks: GSM8K MATH-500 OlympiadBench CSQA SIQA LogiQA AQuA WinoGrande ProofWriter ProntoQA RewardBench ProcessBench

Metrics: accuracy token consumption inference time

Baselines: CoT Self-Consistency Best-of-N BoN Weighted MCTS Beam Search DeepSeek-R1-Distill

Builds on: Best-of-N Monte Carlo Tree Search Self-Consistency Beam Search

Closest papers by meaning

• Bag of Tricks for Inference-time Computation of LLM Reasoning (NeurIPS 2025 · similarity 0.93)
• Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.91)
• Inference-Time Reward Hacking in Large Language Models (NeurIPS 2025 · similarity 0.90)
• Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization (ICLR 2026 · similarity 0.90)
• Know What You Don't Know: Uncertainty Calibration of Process Reward Models (NeurIPS 2025 · similarity 0.90)
• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.90)