Research map / Large language models / Alignment and preferences
HelpSteer 2: Open-source dataset for training top-performing reward models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases HelpSteer2, a CC-BY-4.0 10k-pair multi-attribute helpfulness preference dataset, trains Llama 3 70B and Nemotron-4 340B SteerLM regression reward models (92.0% on RewardBench primary), and proposes SteerLM 2.0 for alignment.
Problem
Permissively licensed preference datasets (Open Assistant, HH-RLHF, HelpSteer) have become less effective for reward modeling, while stronger preference datasets distilled from GPT-4 carry commercial-use restrictions.
Approach
Collect a small but high-quality multi-attribute (helpfulness, correctness, coherence, complexity, verbosity) Likert-scored dataset using responses from permissively usable in-house LLMs, with more annotators per sample and sequential pairwise rating; train SteerLM-style regression reward models on it and use them to align models via Iterative DPO, PPO, and a new SteerLM 2.0 KL-based policy-approx
Key results
- Nemotron-4 340B reward model trained on HelpSteer2 reaches 92.0% overall on RewardBench primary, topping the leaderboard as of June 2024.
- Llama 3 70B reward model trained on HelpSteer2 reaches 88.8%, outperforming same-base models trained on Open Assistant (79.1%) or HH-RLHF (73.9%).
- SteerLM 2.0 Iteration 2 with Daring Anteater reaches MT Bench 8.28, above Llama 3 70B Instruct (8.16) and GPT-4-0613 (8.12).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the Llama 3 70B reward model with HelpSteer2 instead of Open Assistant | Llama 3 70B reward model · RewardBench primary dataset | Overall accuracy: 79.1 -> 88.8 | Helped |
| Train the Llama 3 70B reward model with HelpSteer2 instead of HH-RLHF | Llama 3 70B reward model · RewardBench primary dataset | Overall accuracy: 73.9 -> 88.8 | Helped |
| Scale the reward-model base from Llama 3 70B to Nemotron-4 340B, using HelpSteer2 | HelpSteer2 reward model · RewardBench primary dataset | Overall accuracy: 88.8 -> 92.0 | Helped |
| Replace Daring Anteater with Open Assistant for SFT (ablation) | Llama 3 70B SFT model · MT Bench | GPT-4-Turbo score: 7.96 -> 6.75 | Hurt |
| Use the reward model after SFT on Open Assistant (ablation) | Llama 3 70B · MT Bench | GPT-4-Turbo score: 6.75 -> 7.44 | Helped |
| Use SteerLM 2.0 after the first iteration (ablation) | Llama 3 70B SteerLM model · MT Bench | GPT-4-Turbo score: 8.24 -> 8.28 | Helped |
| Apply DPO using HelpSteer2 preference pairs | Llama 3 70B SFT model · TruthfulQA | MC2: 0.6025 -> 0.6321 | Helped |
| Apply PPO using HelpSteer2 prompts and reward model | Llama 3 70B SFT model · TruthfulQA | MC2: 0.6025 -> 0.5629 | Hurt |
| Apply PPO using HelpSteer2 prompts and reward model | Llama 3 70B SFT model · AlpacaEval 2.0 LC | Length-controlled score: 32.87 (1.40) -> 33.17 (1.38) | Helped |
| Apply SteerLM 2.0 for two iterations rather than the original SteerLM | Llama 3 70B · MT Bench | GPT-4-Turbo score: 8.17 -> 8.28 | Helped |
| Apply SteerLM 2.0 for two iterations rather than the original SteerLM | Llama 3 70B · AlpacaEval 2.0 LC | Length-controlled score: 31.10 (1.37) -> 29.93 (1.35) | Hurt |
| Replace Daring Anteater with Open Assistant for SFT (ablation) | Llama 3 70B SFT model · TruthfulQA | MC2: 0.6025 -> 0.5137 | Hurt |
Surprising findings
- Reward model training on Open Assistant SFT lifted MT Bench from 6.75 to 7.44, a larger gain than from the Daring Anteater SFT data (7.96 to 8.28).
- HelpSteer2-trained models substantially underperform on RewardBench Prior Sets (e.g., 66.5 vs 78.2 for Cohere May 2024), likely due to other models training on those subsets.
Limitations
- English-only and exclusively US-based annotators, limiting cultural and demographic diversity.
- Training done on a best-effort basis, so reported performance may not reflect optimal tuning.
- Attribute combinations are unevenly represented in the dataset; conclusions may not transfer to smaller models.
Experimental setup
Datasets: HelpSteer2 Daring Anteater Open Assistant HH-RLHF ShareGPT HelpSteer RewardBench MT Bench TruthfulQA AlpacaEval 2.0 Arena Hard
Benchmarks: RewardBench MT Bench TruthfulQA MC2 AlpacaEval 2.0 LC Arena Hard
Metrics: accuracy MT Bench score TruthfulQA MC2 AlpacaEval 2.0 length-controlled win rate Arena Hard win rate mean response length inter-annotator quadratic weighted Cohen's kappa
Baselines: Llama 3 70B Instruct GPT-4-0613 ArmoRM-Llama 3 8B RLHFlow-Llama 3 8B Eurus RM Mistral 7B Starling RM Yi 34B Llama 3 70B RM trained on Open Assistant, HH-RLHF, HelpSteer
Builds on: SteerLM HelpSteer Iterative DPO RewardBench
Closest papers by meaning
- HelpSteer2-Preference: Complementing Ratings with Preferences (ICLR 2025 · similarity 0.91)
- HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages (NeurIPS 2025 · similarity 0.90)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.87)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.86)
- Tool-Augmented Reward Modeling (ICLR 2024 · similarity 0.85)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.