Research map / Large language models / Alignment and preferences

Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models

NeurIPS 2025 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Think-RM, which trains a non-reasoning Llama-3.1-8B into a generative reward model with long internal thinking via SFT on long CoT data then rule-based GRPO, plus a pairwise RLHF pipeline that uses pairwise preference strengths directly as advantages.

Problem

Bradley-Terry reward models are sensitive to data size and coverage and vulnerable to reward hacking, while existing generative reward models rely on shallow, few-hundred-token CoT and output pairwise preferences that are incompatible with pointwise RLHF algorithms.

Approach

Model an internal long thinking process in a GenRM (self-reflection, hypothetical and divergent reasoning) using SFT on the longest correct QwQ-32B traces followed by accuracy-only rule-based GRPO, and build a pairwise RLHF advantage estimator from skew-symmetric pairwise preference strengths so policies can be trained without pointwise rewards.

Key results

• Binary Think-RM (SFT+RL) reaches 73.28 average on HelpSteer3-Preference and 86.35 on RewardBench, above BT RM and CoT-GenRM baselines.
• On RM-Bench, Think-RM (SFT) reaches 75.19 binary average vs 68.11 for CoT-GenRM with 16-vote voting, roughly an 8-point gain as claimed.
• Pairwise RLHF with binary Think-RM gives 47.20 WR on AlpacaEval2 vs 33.14 for pointwise RLHF with BT RM.

Measured effects reported in the paper

Surprising findings

• Vertical inference-time scaling with majority voting over 16 judgments gives minimal improvement over single-sample CoT-GenRMs.
• Binary Think-RM's pairwise RLHF length-controlled win rate is nearly identical to CoT-GenRM pairwise RLHF (31.56 vs 31.85), with the gain showing mainly in raw win rate.

Limitations

• Only an 8B backbone was used because larger models are too expensive for the full pairwise RLHF pipeline.
• Multiclass Think-RM underperforms CoT-GenRM (ground truth) on the in-distribution validation set.
• Pairwise RLHF evaluation uses a single relatively easy prompt set (HH-RLHF), and length-controlled win rates are similar to CoT-GenRM.

Experimental setup

Datasets: HelpSteer2-Preference HH-RLHF

Benchmarks: HelpSteer2-Preference HelpSteer3-Preference RewardBench RM-Bench AlpacaEval2

Metrics: preference accuracy length-controlled win rate win rate average response length

Baselines: BT RM CoT-GenRM (ground truth) CoT-GenRM (model-generated) Pointwise RLHF with BT RM

Builds on: GRPO PPO Bradley-Terry reward model CoT-GenRM DeepSeek-R1

Closest papers by meaning

• RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.94)
• Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.91)
• Reward Reasoning Models (NeurIPS 2025 · similarity 0.91)
• Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · similarity 0.91)
• Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models (ICML 2026 · similarity 0.90)
• UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms (ICML 2026 · similarity 0.90)