Research map / Large language models / Reasoning and chain of thought

Rethinking Reward Models for Multi-Domain Test-Time Scaling

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Process reward models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Controlled evaluation of four reward-model variants (discriminative/generative ORM and PRM) for multi-domain best-of-N test-time scaling across 14 domains, finding generative ORM most robust and PRMs weaker outside math.

Problem

Reward models for test-time scaling are mostly studied in math; it is unclear whether PRMs beat ORMs and whether generative verifiers help across broad multi-domain and specialized (medical, legal, science) settings.

Approach

Unified head-to-head comparison of dORM, dPRM, gORM, and gPRM trained under a common protocol with LoRA on R1-Distill backbones, plus theoretical log-error bounds showing PRM error grows with CoT length and analysis of label noise and consensus-filtering length shift.

Key results

• On MMLU-Pro multi-domain outcome verification, dPRM performs on par with dORM while gPRM lags; gORM gives consistent gains across all 14 domains.
• Length-vs-improvement correlations over MV: dPRM -0.058, gPRM -0.608, dORM 0.316, gORM 0.399.
• Wasserstein length-shift distance for gPRM training set is largest in multi-domain (3.083 vs 0.532 for gORM and 0.202 for Train), tied to gPRM degradation (correlation -0.625).

Surprising findings

• dORM performs on par with dPRM in the multi-domain setting, contrary to the math-domain consensus that PRMs are better.
• gPRM is robust to random label noise in math yet degrades in multi-domain settings due to consensus-filtering length shift.

Limitations

• Only supervised fine-tuning; RL-based verifier training not explored.
• LoRA adapters used instead of full-parameter fine-tuning due to compute constraints.
• Tool use not considered in verification.

Experimental setup

Datasets: PRM800K MMLU-Pro ProcessBench GPQA-diamond MedQA LEXam GSM8K MATH

Benchmarks: ProcessBench MMLU-Pro GPQA-diamond MedQA LEXam

Metrics: F1 score best-of-N accuracy improvement over majority voting Wasserstein distance

Baselines: dORM dPRM gPRM majority voting VersaPRM

Builds on: VersaPRM Generative verifiers (ThinkPRM-style gPRM) ProcessBench MMLU-Pro PRM800K

Closest papers by meaning

• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.94)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.93)
• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.92)
• ContextPRM: Leveraging Contextual Coherence for multi-domain Test-Time Scaling (ICLR 2026 · similarity 0.91)
• Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling (ICLR 2026 · similarity 0.91)
• ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs (NeurIPS 2025 · similarity 0.91)