Research map / Reinforcement learning / RL from feedback

Pre-Trained Policy Discriminators are General Reward Models

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes POLAR, a criterion-agnostic reward model pre-training method that trains a policy discriminator with a Bradley-Terry objective on synthetic policy-discrimination data, followed by a 150K-sample SFT stage, yielding POLAR-1.8B and POLAR-7B reward models.

Problem

Reward models for RL post-training of LLMs rely on absolute human preference labels, which are costly, criterion-specific, and generalize poorly; rule-based verifiers do not extend to open-ended tasks.

Approach

Reformulate the reward model as a policy discriminator that scores how similar a candidate trajectory is to a reference trajectory from a target policy; pre-train it on large synthetic data by ranking trajectories from identical policies above those from different policies, then fine-tune with a small human ranking set.

Key results

• POLAR-7B raises STEM preference accuracy from 54.8% to 81.0% and creative writing from 57.9% to 85.5% over the best baselines.
• Llama-3.1-8B average on 20 benchmarks improves from 47.36 to 56.33 with POLAR-7B RFT; Qwen2.5-32B goes from 64.49 to 70.47.
• Power-law fits of validation loss vs. model size and compute reach R^2 of 0.9886 and 0.9912.

Measured effects reported in the paper

Surprising findings

• POLAR-1.8B and POLAR-7B perform similarly on preference benchmarks but POLAR-7B shows a clear advantage in downstream RLHF.
• Providing reference trajectories to baseline reward models often reduces their accuracy, while POLAR benefits from references.

Limitations

• Pre-training requires heavy policy sampling and substantial compute compared with standard LLM data preparation.
• Reliance on reference trajectories raises annotation cost; cross-prompt references are only preliminarily explored.
• Standard preference benchmarks may not reflect downstream RL utility, as shown by the 1.8B vs 7B gap.

Experimental setup

Datasets: RMB UltraFeedback HH-RLHF synthetic policy-sampled pre-training corpus

Benchmarks: RMB OpenCompass 20-benchmark suite in-house real user query preference set

Metrics: preference accuracy average benchmark score validation loss

Baselines: InternLM2-Reward-7B InternLM2-Reward-20B Skywork-Reward-8B Skywork-Reward-27B WorldPM-72B-UltraFeedback Text-Embedding-3-Large similarity

Builds on: Bradley-Terry loss InternLM2.5 PPO GAIL DIRECT

Closest papers by meaning

• Discriminative Policy Optimization for Token-Level Reward Models (ICML 2025 · similarity 0.89)
• RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.88)
• Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.88)
• GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.88)
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.88)
• Learning Goal-Conditioned Representations for Language Reward Models (NeurIPS 2024 · similarity 0.88)