Research map / Large language models / Reasoning and chain of thought
Reward Reasoning Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reward Reasoning Models (RRMs) that generate chain-of-thought before judging pairwise responses, trained with rule-based RL (GRPO) on DeepSeek-R1-distilled Qwen models without reasoning-trace supervision, with ELO and knockout strategies for multi-response reward.
Problem
Reward models use nearly uniform compute per input and cannot adaptively scale test-time compute for complex queries where the correct preference is not immediately apparent.
Approach
Frame reward modeling as a reasoning task: the reward model produces a long chain-of-thought before emitting a pairwise verdict, trained via RL with a rule-based reward (+1 if the ground-truth-preferred response is chosen) so no reasoning traces are needed; multi-response rewards are aggregated via ELO round-robin or knockout tournaments, optionally with majority voting.
Key results
- RRM-32B reaches 91.2 overall on RewardBench (98.6 on Reasoning with voting@16), outperforming DeepSeek-GRM-27B (86.0) and RM-R1-32B (90.9).
- On PPE best-of-32 selection, RRM-32B with voting@5 reaches 79.7 overall versus 56.0 for GPT-4o and 55.8 for Skywork-Reward-Gemma-2-27B.
- DPO on Qwen2.5-7B with RRM-32B labels achieves Arena-Hard 55.4 versus 51.9 with GPT-4o labels and 18.3 for the base model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reward reasoning instead of direct judgment | 32B pairwise reward model · RewardBench Reasoning | accuracy: 90.9 -> 98.3 | Helped |
| Increase RRM scale from 7B to 32B | RewardBench · RewardBench Overall | accuracy: 82.2 -> 91.2 | Helped |
| Add majority voting@16 (ablation) | RRM-32B · RewardBench Overall | accuracy: 91.2 -> 91.9 | Helped |
| Add majority voting@5 (ablation) | RRM-32B knockout best-of-32 · GPQA | accuracy: 61.1 -> 64.3 | Helped |
| Add majority voting@5 (ablation) | RRM-7B knockout best-of-32 · GPQA | accuracy: 49.2 -> 49.0 | Hurt |
| Add majority voting@5 (ablation) | RRM-32B binary preference classifier · Overall (MMLU-Pro, MATH, GPQA) | accuracy: 80.7 -> 81.7 | Helped |
| Use ELO rating instead of knockout tournament (ablation) | RRM-7B and RRM-32B · MATH | accuracy: 88.2 -> 88.5; 90.0 -> 90.3 | Helped |
| Add majority voting to ELO rating (ablation) | RRM-7B and RRM-32B · MATH | accuracy: 88.5 -> 88.8; 90.3 -> 90.5 | Helped |
| Increase the number of pairwise comparisons (ablation) | RRM-7B and RRM-32B · MATH | best-of-N accuracy | Helped |
| Increase the thinking-token budget (ablation) | RRM-7B, RRM-14B, and RRM-32B · RewardBench | accuracy | Helped |
| Use RRM-32B instead of GPT-4o to annotate DPO preferences | Qwen2.5-7B post-training · Arena-Hard | score: 51.9 -> 55.4 | Helped |
| Use RRM-generated ELO rewards for reinforcement learning on unlabeled queries | DeepSeek-R1-Distill-Qwen-7B · MMLU-Pro and GPQA | performance | Helped |
Surprising findings
- Majority voting over reward judgments yields only modest gains in some settings, and for RRM-7B on GPQA best-of-N it slightly reduced accuracy (49.2 to 49.0 on GPQA).
- The paper reports that a DirectJudge model without reasoning trails RRMs mainly on reasoning-heavy categories, showing the gap is attributable to test-time reasoning.
Limitations
- Inputs are restricted to exactly two candidate responses, so multi-response rewards require aggregation schemes.
- Test-time compute scaling increases inference cost due to long reasoning traces.
Experimental setup
Datasets: Skywork-Reward Tulu 3 prompts WebInstruct-verified Skywork-OR1 Big-Math-RL DAPO-Math WebInstruct
Benchmarks: RewardBench PandaLM Test Preference Proxy Evaluations (MMLU-Pro, MATH, GPQA subsets) Arena-Hard MMLU-Pro GPQA MATH
Metrics: accuracy agreement F1 Arena-Hard score
Baselines: Skywork-Reward-Gemma-2-27B GPT-4o Claude 3.5 Sonnet JudgeLM DeepSeek-GRM RM-R1 J1-Llama Qwen2.5-Math-PRM-7B Qwen2.5-Math-PRM-72B
Builds on: GRPO DeepSeek-R1 RewardBench verl
Closest papers by meaning
- RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.94)
- Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.91)
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.90)
- Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling (ICLR 2026 · similarity 0.89)
- Discriminative Policy Optimization for Token-Level Reward Models (ICML 2025 · similarity 0.89)
- Self-Aligned Reward: Towards Effective and Efficient Reasoners (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.