Research map / Reinforcement learning / RL from feedback
Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synthetic study of whether reward-model accuracy predicts downstream policy regret under BoN and PPO, finding a weak-to-moderate correlation and proposing multi-response test sets and a Regressional Goodhart analysis.
Problem
Reward models are evaluated by accuracy on preference test sets, but it is unclear whether accuracy predicts how much policy performance degrades when optimized against the RM (reward overoptimization).
Approach
Builds N=10 Llama-3-8B reward models with different label-flip noise levels, uses one as golden and the rest as proxies to create golden-proxy pairs, and measures policy regret via a Normalised Drop Ratio (NDR); then studies how test-set construction (rank of responses, prompt category, paraphrasing, responses per prompt, annotation budget) changes the accuracy-regret correlation and uses the Regr
Key results
- Accuracy correlates positively with policy regret: Spearman 0.753 under BoN and 0.610 under PPO on the original RewardBench test set.
- Using 5 responses per prompt with xi correlation raises Spearman correlation to 0.677 (BoN) and 0.688 (PPO), versus 0.656 and 0.671 for accuracy on the same 5-response set.
- Under BoN, accuracy-based correlation is weakened when prompt categories differ between RM test and downstream sets (e.g., Math regret vs Chat accuracy Spearman 0.288).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Measure RM accuracy against RewardBench preferences and correlate it with policy regret | Llama-3-Instruct-8B reward models · RewardBench / BoN | Spearman correlation: 0.7533 | Helped |
| Measure RM accuracy against RewardBench preferences and correlate it with policy regret | Llama-3-Instruct-8B reward models · RewardBench / PPO | Spearman correlation: 0.6102 | Helped |
| Sample responses exclusively from the downstream optimization model instead of using original RewardBench responses | RewardBench RM evaluation · RewardBench / BoN | Spearman correlation: 0.753 -> 0.644 | Hurt |
| Sample responses exclusively from the downstream optimization model instead of using original RewardBench responses | RewardBench RM evaluation · RewardBench / PPO | Spearman correlation: 0.610 -> 0.648 | Helped |
| Choose response pairs from rank bins suited to the optimization algorithm | RM accuracy evaluation · RewardBench / BoN and PPO | Correlation with policy regret | Mixed |
| Use an RM test set with prompts matching the downstream test set | RM accuracy evaluation · RewardBench prompt categories / BoN | Spearman correlation | Helped |
| Paraphrase a larger share of test prompts | RM accuracy evaluation · PPO | Spearman correlation | Hurt |
| Increase responses per prompt from two to five (ablation) | Accuracy-based RM evaluation · RewardBench / BoN | Spearman correlation: 0.635±0.039 -> 0.656±0.015 | Helped |
| Increase responses per prompt from two to five (ablation) | Accuracy-based RM evaluation · RewardBench / PPO | Spearman correlation: 0.642±0.047 -> 0.671±0.018 | Helped |
| Use ξ correlation instead of accuracy on five responses per prompt | RM evaluation · RewardBench / BoN | Spearman correlation: 0.656±0.015 -> 0.677±0.026 | Helped |
| Use ξ correlation instead of accuracy on five responses per prompt | RM evaluation · RewardBench / PPO | Spearman correlation: 0.671±0.018 -> 0.688±0.026 | Helped |
| Use ECE instead of accuracy on five responses per prompt | RM evaluation · RewardBench / BoN and PPO | Spearman correlation: 0.656±0.015 -> 0.173±0.016; 0.671±0.018 -> 0.063±0.014 | Hurt |
Surprising findings
- Policies optimized toward RMs with similar accuracy can have quite different regret, and some golden-proxy pairs show overoptimization inconsistent with the Regressional Goodhart prediction.
- Sampling test responses from the downstream optimized model does not consistently improve correlation over the original RewardBench responses.
Limitations
- Synthetic setting uses an RM as the golden reward function rather than real human preferences.
- Correlations are computed from a fixed set of about 2,733 RewardBench prompts, and some results have high variance.
Experimental setup
Datasets: RewardBench Nectar Capybara-7K-binarized Orca-pairs UltraFeedback PKU-SafeRLHF MTBench-human Chatbot-arena HH-RLHF MT-Bench
Benchmarks: RewardBench MT-Bench
Metrics: Spearman correlation Kendall tau Pearson correlation MRR Normalised Drop Ratio pairwise accuracy ECE xi correlation
Baselines: pairwise accuracy ECE Bo5 MRR NDCG
Builds on: RLHF Regressional Goodhart effect Bradley-Terry model RewardBench OpenRLHF
Closest papers by meaning
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.92)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.90)
- Interpreting Language Reward Models via Contrastive Explanations (ICLR 2025 · similarity 0.90)
- Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.90)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.90)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.