Research map / Large language models / Alignment and preferences
HelpSteer2-Preference: Complementing Ratings with Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases HelpSteer2-Preference (preference annotations with human justifications), compares Bradley-Terry and SteerLM regression reward models on matched data, proposes Scaled BT initialized from a helpfulness regression model plus ExPO, and uses the resulting reward model for DPO/PPO/REINFORCE alignment of Llama-3.1-70B-Instruct.
Problem
Reward model training has two paradigms (Bradley-Terry and regression) but no public data matched for both formats, so it is unclear which is better; existing data is either collected for one format or too noisy.
Approach
Collect 7-point preference annotations with human-written justifications on the same HelpSteer2 prompt-response pairs as Likert ratings, enabling a matched head-to-head comparison; propose Scaled Bradley-Terry loss that multiplies the BT loss by preference magnitude, initialized from a helpfulness-only regression reward model and combined with ExPO weight extrapolation.
Key results
- Scaled BT initialized from helpfulness-only SteerLM regression plus ExPO reaches 94.1 overall on RewardBench, top of the leaderboard as of 1 Oct 2024.
- REINFORCE aligned Llama-3.1-70B-Instruct reaches 85.0 on Arena Hard, versus 55.7 for the base Llama-3.1-70B-Instruct and 79.3 for GPT-4o.
- Helpfulness-only SteerLM regression (93.0) and best Bradley-Terry (93.7 without ExPO) perform comparably, and Pairwise Justifier models score at most 90.0 on RewardBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize Scaled BT with Helpfulness-Only Regression and apply ExPO (ablation) | Scaled BT reward model · RewardBench | overall accuracy: 93.7 -> 94.1 | Helped |
| Train BT from scratch with Scaled BT loss rather than Regular BT or Margin BT (ablation) | Bradley-Terry reward model · RewardBench | overall accuracy: 91.5 -> 92.7 | Helped |
| Initialize Scaled BT with Helpfulness-Only Regression (ablation) | Bradley-Terry reward model · RewardBench | overall accuracy: 92.7 -> 93.7 | Helped |
| Train Regression on helpfulness only rather than all five HelpSteer attributes (ablation) | SteerLM Regression · RewardBench | overall accuracy: 92.4 -> 93.0 | Helped |
| Use up to three preference justifications per task rather than one (ablation) | Pairwise Justifier · RewardBench | overall accuracy: 86.1 to 88.1 | Helped |
| Flip response order and labels as data augmentation (ablation) | Pairwise Justifier · RewardBench | overall accuracy: +1.3% | Helped |
| Remove preference elaborations while retaining preference statements (ablation) | Pairwise Justifier · RewardBench | overall accuracy: 88.4 vs 88.1 | Hurt |
| Use only Response 1 or 2 as the label instead of preference justifications (ablation) | Pairwise Justifier · RewardBench | overall accuracy: 88.1 -> 90.0 | Helped |
| Train with Scaled DPO rather than Regular or Margin DPO (ablation) | DPO alignment · MT Bench | GPT-4-Turbo score: 8.66 / 8.58 -> 8.74 | Helped |
| Use online REINFORCE rather than PPO | Online RLHF alignment · Arena Hard | score: 58.6 -> 85.0 | Helped |
| Use online REINFORCE rather than PPO | Online RLHF alignment · MT Bench | GPT-4-Turbo score: 8.74 -> 8.98 | Helped |
| Use Online RLHF rather than DPO | Alignment with HelpSteer2-Preference · AlpacaEval 2.0 LC | score: 41.0 -> 43.8 / 57.6 | Helped |
Surprising findings
- Training the pairwise justifier on preference labels alone outperformed training on full human-written justifications (90.0 vs 88.1 overall).
- Best reward model trails Skywork-Reward-Gemma-2-27B by a large margin on Chat-Hard (85.7 vs 91.4), mainly on GPT-4-judged subsets.
Limitations
- Preference dataset is small (about 7k pairs) and general-domain only, so conclusions may not transfer to other domains or sizes.
- Automated evaluation relies on GPT-4-Turbo as judge, which may be weaker than frontier models and may not fully assess aligned model quality.
- Best reward model underperforms on Chat-Hard subsets that use GPT-4 judgments, possibly due to GPT-4 bias in training data of competitors.
Experimental setup
Datasets: HelpSteer2 HelpSteer2-Preference RewardBench
Benchmarks: RewardBench Arena Hard AlpacaEval 2.0 LC MT Bench
Metrics: RewardBench accuracy (overall and per category) Arena Hard win rate AlpacaEval 2.0 length-controlled win rate MT Bench score mean response length
Baselines: Skywork-Reward-Gemma-2-27B TextEval-Llama3.1-70B Skywork-Critic-Llama-3.1-70B SFR-LLaMa-3.1-70B-Judge-r Nemotron-4-340B-Reward GPT-4o-2024-08-06 Claude-3.5-Sonnet-20240620 Llama-3.1-405B-Instruct PPO Regular DPO
Builds on: SteerLM HelpSteer2 Bradley-Terry DPO ExPO RewardBench
Closest papers by meaning
- HelpSteer 2: Open-source dataset for training top-performing reward models (NeurIPS 2024 · similarity 0.91)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.85)
- Diverging Preferences: When do Annotators Disagree and do Models Know? (ICML 2025 · similarity 0.85)
- Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment (ICML 2025 · similarity 0.85)
- HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages (NeurIPS 2025 · similarity 0.84)
- Tool-Augmented Reward Modeling (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.