Research map / Large language models / Reasoning and chain of thought
ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReasonFlux-PRM, a trajectory-aware process reward model that scores both thinking trajectories and final responses of long-CoT data, and applies it to offline data selection, GRPO reward shaping, and best-of-N test-time scaling.
Problem
Existing process reward models are trained on final responses and fail to reliably evaluate the intermediate thinking trajectories produced by frontier reasoning models (trajectory-response format), degrading downstream SFT on such data.
Approach
A PRM trained with a joint objective combining step-level rewards (softmax-weighted mix of embedding alignment to the final response, LLM-judge quality, and contrastive coherence between adjacent steps) and a trajectory-level reward from template-guided policy solve rates, then used offline for data selection, online in GRPO, and for best-of-N.
Key results
- ReasonFlux-PRM-7B-selected 1k SFT data lifts Qwen2.5-14B-Instruct MATH500 by 6.0% over s1k human-curated data and GPQA-Diamond by 6.1%.
- GRPO with ReasonFlux-PRM-7B on DeepSeek-R1-Distill-Qwen-7B raises MATH500 from 89.6 (rule-based) to 94.8 and GPQA-Diamond from 47.1 to 51.6.
- On Qwen2.5-7B-Instruct, ReasonFlux-PRM-7B improves AIME24 from 12.9 to 16.3 and AIME25 from 11.1 to 17.1 relative to rule-based GRPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReasonFlux-PRM-7B instead of Qwen2.5-Math-PRM-7B as GRPO reward | Qwen2.5-7B-Instruct · AIME24 | accuracy: 12.9 -> 16.3 | Helped |
| Use ReasonFlux-PRM-7B instead of Qwen2.5-Math-PRM-7B as GRPO reward | Qwen2.5-7B-Instruct · AIME25 | accuracy: 13.3 -> 17.1 | Helped |
| Use ReasonFlux-PRM-7B instead of Qwen2.5-Math-PRM-7B as GRPO reward | Qwen2.5-7B-Instruct · MATH500 | accuracy: 74.8 -> 77.2 | Helped |
| Use ReasonFlux-PRM-7B instead of Qwen2.5-Math-PRM-7B as GRPO reward | Qwen2.5-7B-Instruct · GPQA-Diamond | accuracy: 32.4 -> 34.9 | Helped |
| Use ReasonFlux-PRM-7B instead of Qwen2.5-Math-PRM-7B as GRPO reward | DeepSeek-R1-Distill-Qwen-7B · MATH500 | accuracy: 92.8 -> 94.8 | Helped |
| Use ReasonFlux-PRM-7B instead of Qwen2.5-Math-PRM-7B as GRPO reward | DeepSeek-R1-Distill-Qwen-7B · GPQA-Diamond | accuracy: 49.1 -> 51.6 | Helped |
| Increase trajectory-level reward weight alpha from 0.1 to 0.8 (ablation) | ReasonFlux-PRM-7B offline selection · AIME24 | accuracy: 26.7 -> 40.0 | Helped |
| Increase trajectory-level reward weight alpha from 0.1 to 0.8 (ablation) | ReasonFlux-PRM-7B offline selection · MATH500 | accuracy: 81.2 -> 83.6 | Helped |
| Increase process-reward weight beta from 0.1 to 0.8 (ablation) | Qwen2.5-7B-Instruct GRPO · AIME24 | accuracy: 10.0 -> 20.0 | Helped |
| Increase process-reward weight beta from 0.1 to 0.8 (ablation) | Qwen2.5-7B-Instruct GRPO · AIME25 | accuracy: 6.7 -> 16.7 | Helped |
| Increase process-reward weight beta from 0.1 to 0.8 (ablation) | Qwen2.5-7B-Instruct GRPO · MATH500 | accuracy: 73.6 -> 76.8 | Helped |
Surprising findings
- Existing PRMs (including Qwen2.5-Math-PRM-72B) selecting SFT data from trajectory-response traces underperform random 1k selection on several benchmarks, e.g., AIME25 drops.
- Scoring only the final response instead of the thinking trajectory gives clearer separation between oracle models' data.
Limitations
- Relies on GPT-4o as judge for quality and template scores, adding cost and dependence on a proprietary model.
- Alpha and beta weights are hand-tuned; the authors note the optimal value may depend on data distribution.
- Evaluation on a small set of benchmarks (AIME with 30 problems, GPQA-Diamond) with high variance.
Experimental setup
Datasets: OpenThoughts-114K s1k s1 (59K Gemini traces) MATH DAPO NuminaMath-CoT
Benchmarks: AIME24 AIME25 MATH500 GPQA-Diamond
Metrics: accuracy mean accuracy over 16 runs
Baselines: Math-Shepherd-PRM-7B Skywork-PRM-7B Qwen2.5-Math-PRM-7B Qwen2.5-Math-PRM-72B s1k human-curated data rule-based GRPO
Builds on: GRPO Math-Shepherd Qwen2.5-Math-PRM s1 ReasonFlux
Closest papers by meaning
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.92)
- Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.91)
- Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.91)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
- Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · similarity 0.90)
- Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.