Research map / Large language models / Reasoning and chain of thought

From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Process reward models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Empirically studies process reward models (Qwen2.5-based PRMs) for scaling, training data diversity, test-time search strategies, and math-to-code generalization, proposing ASLAF step-level annotation and filtering.

Problem

How PRM training compute, dataset diversity, and test-time search strategies affect PRM accuracy, and whether PRMs trained on math transfer to code generation.

Approach

Introduces ASLAF, an automatic step-level annotation and ensemble-filtering pipeline combining PRM800K and Math-Shepherd, and compares PRMs trained on math vs. code data across BoN, beam search, MCTS, and majority voting, plus a gradient-based pattern-similarity metric.

Key results

• On code benchmarks with Qwen2.5-Coder-32B-Instruct, PRM-Math (best-of-8) reaches 91.5 on HumanEval+ vs 82.9 for single sample and 85.8 for majority vote.
• With Qwen2.5-Coder-7B-Instruct, PRM-Math raises LiveCodeBench from 14.2 to 22.6 at N=8.
• The paper reports MCTS as the most effective search under ample compute and best-of-N as best under tight time budgets, based on figures.

Surprising findings

• PRMs trained on math data performed comparably to or better than PRMs trained on code data on code generation benchmarks.
• Code-trained PRM (PRM-Code) did not show a significant advantage over math-trained PRM on code tasks.

Limitations

• Compute constraints limited exploration of larger models and more complex scaling strategies.
• Dataset diversity scope was limited; generalization beyond math and code untested.
• Evaluation focused on accuracy and efficiency, without interpretability or robustness metrics.

Experimental setup

Datasets: PRM800K Math-Shepherd ASLAF TACO APPS CodeContent MATH-500 AIME HumanEval+ MBPP+ LiveCodeBench BigCodeBench

Benchmarks: MATH-500 AIME PRM800K HumanEval+ MBPP+ LiveCodeBench BigCodeBench

Metrics: accuracy pass@N maj@N prm@N

Baselines: Majority Vote Beam Search Best-of-N

Builds on: Math-Shepherd PRM800K Qwen2.5

Closest papers by meaning

• Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.94)
• GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning (AAAI 2026 · similarity 0.93)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.92)
• AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence (ICML 2025 · similarity 0.92)
• Know What You Don't Know: Uncertainty Calibration of Process Reward Models (NeurIPS 2025 · similarity 0.92)
• ContextPRM: Leveraging Contextual Coherence for multi-domain Test-Time Scaling (ICLR 2026 · similarity 0.91)