Research map / Large language models / Reasoning and chain of thought
VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows open-source math PRMs fail to generalize beyond math, then builds VersaPRM by fine-tuning a Llama-based math PRM on a synthetic multi-domain MMLU-Pro CoT dataset with LLM-generated, LLM-labeled step labels.
Problem
Process reward models are trained mainly on math data and their effectiveness in non-mathematical domains (law, philosophy, biology, etc.) is unclear and appears poor for test-time reranking and search.
Approach
Generate step-wise CoTs for MMLU-Pro questions with Llama-3.1-8B-Instruct and auto-label each step with Llama-3.1-70B-Instruct using step definitions and the ground-truth answer, then fine-tune a math PRM (LlamaPRM800K) with LoRA on this multi-domain data to yield VersaPRM.
Key results
- On MMLU-Pro-CoT-Eval, VersaPRM with WMV reaches 61.22 overall vs 57.15 for MV (+4.07), compared with +1.02 for Qwen-2.5-Math-PRM.
- In the Law category VersaPRM gains +7.93 over MV (43.86 vs 35.93), while Qwen-2.5-Math-PRM gains +0.35.
- Manual check of 64 auto-labeled questions gave 80% agreement for correct-labeled CoTs and 71% for incorrect-labeled CoTs; ~75% of data estimated correctly labeled, at under $100 cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune a math PRM on synthetic multi-domain reasoning data | LlamaPRM800K · MMLU-Pro-CoT-Eval (Unlabeled), Law | accuracy: 35.93 -> 43.86 | Helped |
| fine-tune a math PRM on synthetic multi-domain reasoning data | LlamaPRM800K · MMLU-Pro-CoT-Eval (Unlabeled), All except math | accuracy: 56.61 -> 60.29 | Helped |
| train on multi-domain CoTs instead of a same-size math-only subset (ablation) | LlamaPRM800K · MMLU-Pro-CoT-Eval (Unlabeled), Math and non-Math | accuracy | Helped |
| hold out one domain (Law or Biology) during training (ablation) | VersaPRM · MMLU-Pro-CoT-Eval (Unlabeled), held-out Law and Biology | accuracy | No clear effect |
| initialize from Llama-3.1-8B-Instruct instead of LlamaPRM800K (ablation) | VersaPRM · MMLU-Pro-CoT-Eval (Unlabeled), Math | accuracy | Hurt |
| initialize from QwenPRM800K instead of LlamaPRM800K (ablation) | VersaPRM · MMLU-Pro-CoT-Eval (Unlabeled), Math and non-Math | accuracy | Hurt |
| use full fine-tuning instead of LoRA (ablation) | VersaPRM initialized from LlamaPRM800K · MMLU-Pro-CoT-Eval (Unlabeled), Math and non-Math | accuracy | Hurt |
| remove step-label definitions and the ground-truth answer from the auto-labeling prompt (ablation) | VersaPRM data-generation pipeline · MMLU-Pro-CoT-Train (Labeled), correct-CoT labeling agreement | agreement rate: over 70% ... to 7% | Hurt |
| omit the ground-truth answer from the auto-labeling prompt (ablation) | VersaPRM · MMLU-Pro-CoT-Eval (Unlabeled), aggregate BoN and WMV | accuracy | Hurt |
| add counterfactual augmentation to training data (ablation) | VersaPRM · MMLU-Pro-CoT-Eval (Unlabeled), aggregate BoN and WMV | accuracy | No clear effect |
| add rewrite augmentation to training data (ablation) | VersaPRM · MMLU-Pro-CoT-Eval (Unlabeled), aggregate BoN and WMV | accuracy | No clear effect |
| self-filter labeled training CoTs (ablation) | VersaPRM · MMLU-Pro-CoT-Eval (Unlabeled), aggregate BoN and WMV | accuracy | No clear effect |
Surprising findings
- Removing the system prompt with step definitions and the ground-truth answer from the auto-labeling prompt drops agreement on correct-labeled CoTs from 80% to 7%.
- Training on a random multi-domain subset outperforms training on a same-sized math-only subset even in the math domain.
Limitations
- Auto-labels are estimated to be only ~75% accurate, relying on Llama-family generator and labeler.
- Evaluation uses Llama-3.1-8B-Instruct generated CoTs and MMLU-Pro multiple-choice questions, not open-ended problems.
- Counterfactual and rewrite augmentations and self-filtering gave no significant benefit.
Experimental setup
Datasets: MMLU-Pro MATH GSM8K PRM800K
Benchmarks: MMLU-Pro-CoT-Eval (Unlabeled) MMLU-Pro
Metrics: accuracy gain over majority voting
Baselines: Math-Shepherd Qwen-2.5-Math-PRM Math-PSA RLHFLow-Deepseek LlamaPRM800K QwenPRM800K Llama-3.1-70B-Instruct as judge Majority Voting
Builds on: Math-Shepherd PRM800K MMLU-Pro LoRA
Closest papers by meaning
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.91)
- ContextPRM: Leveraging Contextual Coherence for multi-domain Test-Time Scaling (ICLR 2026 · similarity 0.90)
- GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning (AAAI 2026 · similarity 0.90)
- Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.89)
- Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning (AAAI 2026 · similarity 0.89)
- AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.