Research map / Reinforcement learning / RL from feedback
What Makes a Reward Model a Good Teacher? An Optimization Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that reward models must induce sufficient reward variance (not just accuracy) for efficient RLHF policy-gradient optimization, so accurate reward models can be poor teachers and reward quality depends on the policy.
Problem
Reward models for RLHF are evaluated mainly by accuracy, but it is unclear whether accuracy captures what makes a reward model an effective teacher during policy-gradient optimization.
Approach
Analyzes RLHF optimization via gradient flow and shows the time to increase expected reward is lower-bounded by inverse average reward variance under the initial policy, so low-variance reward models yield flat landscapes regardless of accuracy; then proves perfectly accurate low-variance reward models can be slower than inaccurate ones and that reward model rankings depend on the initial policy.
Key results
- A perfectly accurate reward model with low reward variance (0.111) underperforms less accurate models in ground truth reward increase, while reward variance correlates strongly with proxy reward increase (Spearman 1.000).
- Combined reward variance and accuracy measure achieves Pearson 0.940 with ground truth reward increase, versus -0.283 for on-policy accuracy alone.
- For different initial policies, the best reward model varies; RM-Gemma-2B induces highest reward variance for all policies but lower ground truth reward due to low accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a reward model that induces higher reward variance for the initial policy | RLOO policy gradient with Pythia-2.8B · UltraFeedback | ground truth reward increase: 0.834 Pearson correlation; 0.714 Spearman correlation | Helped |
| Use reward variance together with accuracy to assess reward models | RLOO policy gradient with Pythia-2.8B · UltraFeedback | ground truth reward increase: 0.940 Pearson correlation; 0.828 Spearman correlation | Helped |
| Rely on accuracy alone to predict optimization effectiveness | RLOO policy gradient with Pythia-2.8B · UltraFeedback | ground truth reward increase: -0.283 Pearson correlation; 0.200 Spearman correlation | No clear effect |
| Reduce reward separation while preserving rankings, producing perfect accuracy but low reward variance | RLOO policy gradient with Pythia-2.8B · UltraFeedback | ground truth reward increase | Hurt |
| Use a proxy reward model instead of directly optimizing the ground truth reward | RLOO policy gradient with Pythia-2.8B · UltraFeedback, first few epochs | ground truth reward increase | Helped |
| Use a reward model with high off-policy accuracy but low on-policy accuracy | RLHF policy gradient · UltraFeedback | ground truth reward increase | Hurt |
| Match the reward model to the initial language model instead of using one universal reward model | Pythia and Llama-3.2 policies · UltraFeedback | ground truth reward increase | Helped |
| Use RM-Gemma-2B, which induces the highest reward variance | Different initial language models · UltraFeedback | proxy reward increase | Helped |
| Use RM-Gemma-2B despite its inaccuracy | Different initial language models · UltraFeedback | ground truth reward increase | No clear effect |
Surprising findings
- A perfectly accurate reward model can yield slower ground truth reward increase than substantially inaccurate reward models due to low reward variance.
- Even when the ground truth reward is accessible, a proxy reward model can be more effective in early training epochs.
Limitations
- Theory uses gradient flow (small learning rate limit) and exact gradients rather than sample-based estimates.
- Main theoretical separations for tabular policies; general autoregressive results have Jacobian-dependent constants.
- Reward variance improvement via scaling is not explored beyond preliminary experiments.
Experimental setup
Datasets: UltraFeedback AlpacaFarm
Benchmarks: UltraFeedback test prompts (reported as nearly identical to training set)
Metrics: reward variance on-policy accuracy off-policy accuracy proxy reward increase ground truth reward increase Pearson correlation Spearman correlation
Baselines: accuracy-based reward models trained on on-policy/off-policy preference data ground truth reward (ArmoRM) GRM-Llama-3.2-3B GRM-Gemma-2-2B RM-Tulu-V2-8B RM-Gemma-2B
Builds on: RLOO PPO GRPO Bradley-Terry reward modeling
Closest papers by meaning
- On the Robustness of Reward Models for Language Model Alignment (ICML 2025 · similarity 0.89)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.89)
- The Trickle-down Impact of Reward Inconsistency on RLHF (ICLR 2024 · similarity 0.89)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.89)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.89)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.