Research map / Large language models / Reasoning and chain of thought

RM-R1: Reward Modeling as Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Reasoning Reward Models (ReasRMs) that cast reward modeling as reasoning, training RM-R1 (7B-32B) via reasoning distillation followed by GRPO-based RLVR with a chain-of-rubrics rollout; reports SOTA average across RewardBench, RM-Bench, and RMB.

Problem

Generative and scalar reward models give opaque or superficial judgments, which limits their accuracy on reasoning-intensive preference tasks.

Approach

Treat the reward model as a reasoning policy: distill high-quality reasoning traces, then apply RL with verifiable correctness rewards, using a chain-of-rubrics rollout that classifies queries as chat or reasoning and generates rubrics or self-solutions before judging.

Key results

• RM-R1-DeepSeek-Distilled-Qwen-32B reaches 81.5 average over RewardBench, RM-Bench, and RMB, and 83.9 on RM-Bench, the best reported there.
• RM-R1-DeepSeek-Distilled-Qwen-14B averages 79.6, surpassing INF-ORM-Llama3.1-70B (78.8) and Nemotron-4-340B-Reward (77.1) on average.
• Ablation on RewardBench: Cold Start RL 89.5 average, adding rubrics and query categorization raises it to 90.8, and full distillation+RL reaches 91.4 (Qwen-2.5-Instruct-32B base).

Surprising findings

• Pure RL from an instruct model (cold start) overfits to superficial features and underperforms the distilled-then-RL pipeline.
• Larger scalar RMs on RewardBench often do not beat smaller ones on the other benchmarks, suggesting RewardBench alone is an incomplete assessment.

Limitations

• Distillation traces depend on proprietary oracle models (Claude-3.7-Sonnet, o3), and about 25% of Claude traces were incorrect and needed correction.
• Reward is purely correctness-based; format reward gave no significant gain, and the method is evaluated only on pairwise text preference benchmarks.
• Some baseline numbers are taken from tech reports rather than reproduced; the paper notes potential data contamination for some baselines.

Experimental setup

Datasets: Skywork-Reward-Preference-80K-v0.2 (cleaned) Code-Preference-Pairs (8K subset) Math-Step-DPO-10K RewardBench RM-Bench RMB

Benchmarks: RewardBench RM-Bench RMB

Metrics: accuracy average accuracy across benchmarks

Baselines: INF-ORM-Llama3.1-70B Nemotron-4-340B-Reward GPT-4o-0806 Claude-3-5-sonnet-20240620 Self-taught-evaluator-llama3.1-70B DeepSeek-GRM-27B JudgeLRM Skywork-Reward-Gemma-2-27B

Builds on: GRPO DeepSeek-R1 Self-taught evaluators DeepSeek-GRM JudgeLRM

Closest papers by meaning

• Reward Reasoning Models (NeurIPS 2025 · similarity 0.94)
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.94)
• UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms (ICML 2026 · similarity 0.92)
• Step-wise Rubric Rewards for LLM Reasoning (NeurIPS 2026 · similarity 0.91)
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.91)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.91)