Research map / Large language models / Alignment and preferences
HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases HelpSteer3-Preference, a CC-BY-4.0 human-annotated preference dataset of ~40k samples across General, STEM, Code and Multilingual tasks, and trains Bradley-Terry and generative reward models plus RLOO-aligned policies from it.
Problem
Existing general-domain preference datasets are either low quality, English-only, lack diversity in challenging tasks, or carry commercial-use restrictions (e.g., GPT-4 annotations), limiting RLHF research.
Approach
Collect high-quality human preference annotations from specialist annotator pools (STEM degree holders, software engineers, language-fluent annotators) on diverse prompts from WildChat and ShareGPT, with responses from 17 commercially permissive models, then train RMs and GenRMs on the data.
Key results
- Multilingual-subset Bradley-Terry RM reaches 82.4% on RM-Bench overall, versus prior best 70.1%.
- English RM (General+STEM+Code) reaches 73.7% on JudgeBench, versus prior best 64.3%.
- Generative RM with voting@32 reaches 85.5% on RM-Bench and 78.6% on JudgeBench.
- RLOO with English RM raises Llama-3.3-70B-Instruct Arena Hard from 62.4 to 87.0 and WildBench from 52.5 to 60.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a Bradley-Terry RM on the HelpSteer3-Preference Multilingual subset instead of English subsets | Llama-3.3-70B-Instruct reward model · RM-Bench | accuracy: 79.9 -> 82.4 | Helped |
| Train on General, STEM, and Code subsets instead of the Multilingual subset | Llama-3.3-70B-Instruct reward model · JudgeBench | accuracy: 69.4 -> 73.7 | Helped |
| Use HelpSteer3-Preference rather than the strongest baseline preference dataset | Bradley-Terry reward model · RM-Bench | accuracy: 77.3 -> 82.4 | Helped |
| Use HelpSteer3-Preference rather than the strongest baseline preference dataset | Bradley-Terry reward model · JudgeBench | accuracy: 68.6 -> 73.7 | Helped |
| Use generative reward modeling instead of Bradley-Terry modeling with the best respective HelpSteer3 subsets | Reward model · RM-Bench | accuracy: 82.4% to 84.2% | Helped |
| Use generative reward modeling instead of Bradley-Terry modeling with the best respective HelpSteer3 subsets | Reward model · JudgeBench | accuracy: 73.7% to 75.1% | Helped |
| Apply voting@32 to the generative RM | Generative RM trained on all HelpSteer3-Preference subsets · RM-Bench | accuracy: 84.2% to 85.5% | Helped |
| Apply voting@32 to the generative RM | Generative RM trained on all HelpSteer3-Preference subsets · JudgeBench | accuracy: 75.1% to 78.6% | Helped |
| Use all subsets instead of General + STEM + Code (ablation) | Bradley-Terry reward model · RM-Bench | accuracy: 79.9 -> 78.5 | Hurt |
| Exclude the STEM subset from training (ablation) | Bradley-Terry reward model · RM-Bench | accuracy: 78.5 -> 77.1 | Hurt |
| Exclude the Code subset from training (ablation) | Bradley-Terry reward model · JudgeBench | accuracy: 68.9 -> 70.9 | Helped |
| Train on the Code subset only rather than all HelpSteer3-Preference subsets (ablation) | Bradley-Terry reward model · RM-Bench | accuracy: 78.5 -> 73.5 | Hurt |
Surprising findings
- Training only on the Code subset yields poor code performance on RM-Bench and JudgeBench (54.8% on JudgeBench coding), attributed to differing benchmark code sources.
- Combining English and Multilingual subsets reduces overall RM-Bench and JudgeBench performance relative to individual subsets.
Limitations
- No multimodal (vision) samples due to licensing constraints.
- Only 14 programming and 13 natural languages; no cross-domain intersections such as non-English code prompts.
- Multi-turn prompt in-filling can occasionally produce unnatural conversations.
Experimental setup
Datasets: HelpSteer3-Preference HelpSteer2-Preference Skywork-Preference INF-ORM-Preference WildChat-1M ShareGPT
Benchmarks: RM-Bench JudgeBench MT Bench Arena Hard WildBench
Metrics: accuracy overall score win rate weighted Cohen's kappa
Baselines: Llama-3.1-Nemotron-70B-Reward Skywork-Reward-Gemma-2-27B Skywork-Reward-Llama-3.1-8B HelpSteer2-Preference INF-ORM-Preference Skywork-Preference gpt-4o-2024-05-13 Claude-3.5-Sonnet-20240620
Builds on: HelpSteer2-Preference HelpSteer3 Feedback RM-Bench JudgeBench DeepSeek-GRM
Closest papers by meaning
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.91)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.90)
- Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment (ICML 2025 · similarity 0.90)
- Tool-Augmented Reward Modeling (ICLR 2024 · similarity 0.90)
- HelpSteer 2: Open-source dataset for training top-performing reward models (NeurIPS 2024 · similarity 0.90)
- Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.