Research map / Large language models / Alignment and preferences
How to Evaluate Reward Models for RLHF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Preference Proxy Evaluations (PPE), a reward model benchmark combining crowdsourced Chatbot Arena human preferences and verifiable correctness preferences, validated by correlating reward model metrics with Arena scores of DPO-trained Llama-3.1-8B models.
Problem
Reward model evaluation is typically not linked to downstream RLHF outcomes, and running full RLHF to test each reward model is prohibitively expensive.
Approach
Build a reward model benchmark from real crowdsourced human preferences and verifiable correctness labels on sampled responses, then empirically validate which metrics correlate with post-RLHF human preference by DPO-training Llama-3.1-8B-Instruct with nine reward models and deploying the models on Chatbot Arena.
Key results
- Accuracy on the human preference dataset correlates best with post-DPO Arena scores, with Pearson correlation up to about 0.80 at low quantile aggregation.
- Overall evaluation achieves a 77% Pearson correlation with downstream performance, per the conclusion.
- Across 12,190 Chatbot Arena votes, Athene-RM-70B-trained DPO model scored 1216 Arena score, second only to Llama-3.1-70B-Instruct at 1228 among non-baseline-level comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pairwise human-preference accuracy rather than aggregate ranking-correlation metrics to predict DPO outcomes | Reward-model evaluation on the PPE human preference dataset · Post-DPO Chatbot Arena | Pearson correlation with Arena Score: 0.80 | Helped |
| Evaluate correctness using ROC AUC rather than accuracy | Reward-model evaluation on correctness benchmarks · MMLU-Pro, MATH, GPQA, MBPP-Plus, and IFEval | correlation with post-DPO Arena Score | Helped |
| Use Math as the individual correctness-evaluation domain | Reward-model evaluation on correctness benchmarks · Post-DPO Chatbot Arena | correlation with Arena Score | Helped |
| Aggregate human-preference domain scores toward the lower quantile rather than the average or upper quantile | PPE human preference metrics · Post-DPO Chatbot Arena | Pearson correlation with Arena Score | Helped |
| Use RewardBench evaluation score as a proxy for post-RLHF performance on top models | Reward models · Post-RLHF human preference | correlation with downstream performance | Hurt |
| Measure granular pairwise accuracy rather than aggregate Spearman or Kendall correlation | Reward-model metrics on human preferences · Post-DPO Chatbot Arena | correlation with downstream Arena Score | Helped |
Surprising findings
- Spearman and Kendall correlation metrics have nearly zero correlation with post-DPO Arena scores despite being common.
- RewardBench score shows a negative correlation with downstream RLHF performance for top models.
Limitations
- Only nine reward models could be tested end-to-end due to cost.
- Downstream evaluation uses offline DPO on a single base model, not online PPO.
- Possible benchmark leakage from public datasets.
Experimental setup
Datasets: Chatbot Arena human preference battles MMLU-Pro MATH GPQA MBPP Plus IFEval
Benchmarks: PPE RewardBench MMLU-Pro MATH GPQA MBPP-Plus IFEval Chatbot Arena
Metrics: accuracy Spearman correlation Kendall correlation row-wise Pearson correlation separability confidence agreement Brier score best-of-K maximum AUC Pearson correlation with post-RLHF Arena score
Baselines: RewardBench NaiveVerbosityModel Ensemble-Judges
Builds on: RewardBench Arena-Hard-Auto DPO
Closest papers by meaning
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.93)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.93)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.92)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.92)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.92)
- Avoiding exp(R) scaling in RLHF through Preference-based Exploration (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.