Research map / Large language models / Alignment and preferences
Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how RLHF prompt-data construction limits data scaling, proposing Pre-PPO prompt selection (choosing low-reward prompts) and early-stage math/coding prioritization, evaluated on internal ~25B and ~150B models.
Problem
Naively scaling RLHF training prompts (from 1M to 6M) yields no gains in PPO-based RLHF due to reward hacking and loss of response diversity; prompt construction and its scalability is under-studied.
Approach
Select training prompts with the lowest generative reward model scores (Pre-PPO) so they are harder and less prone to reward hacking, and train on math and coding prompts first, since their ground-truth-verified rewards resist hacking and preserve fine-grained response distinctions.
Key results
- On the large model (V2.0), the combined method improves overall score from 34.0 to 35.4 (+1.4) and STEM by +3.9 over the PPO baseline.
- Human evaluation on the large model shows overall +3.2 (64.4 to 67.6, p=0.01).
- Increasing the Pre-PPO selected newly collected data from 10% to 20% or 50% degraded RLHF performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine Pre-PPO prompt selection with early math/coding emphasis | PPO-based RLHF · TestSet V1.0, small and large models | Overall: 37.7 -> 38.8; 49.7 -> 50.8 | Helped |
| Combine Pre-PPO prompt selection with early math/coding emphasis | PPO-based RLHF · TestSet V2.0, small and large models | Overall: 23.9 -> 25.1; 34.0 -> 35.4 | Helped |
| Combine Pre-PPO prompt selection with early math/coding emphasis | PPO-based RLHF · Human evaluation | Overall: 64.4 -> 67.6 | Helped |
| Use Pre-PPO prompt selection without early math/coding emphasis (ablation) | PPO-based RLHF · TestSet V2.0 | Overall: 34.0 -> 35.1 | Helped |
| Add early math/coding emphasis to Pre-PPO (ablation) | Pre-PPO-Large · TestSet V2.0 | Overall: 35.1 -> 35.4 | Helped |
| Add early math/coding emphasis to Pre-PPO (ablation) | Pre-PPO-Large · TestSet V2.0 | STEM: 30.8 -> 31.9 | Helped |
| Add early math/coding emphasis to Pre-PPO (ablation) | Pre-PPO-Large · TestSet V2.0 | Coding: 49.5 -> 50.7 | Helped |
| Add early math/coding emphasis to Pre-PPO (ablation) | Pre-PPO-Large · TestSet V2.0 | OOD: 37.9 -> 37.1 | Hurt |
| Increase newly collected training data from 10% to 20% or 50% (ablation) | Pre-PPO · RLHF evaluation | Overall performance | Hurt |
| Prioritize math and coding prompts during early RLHF training (ablation) | PPO-based RLHF · TestSet V1.0 | Coding and STEM performance | Helped |
| Select lower-scoring, more challenging prompts with Pre-PPO (ablation) | PPO-based RLHF · Training steps 2000–4000 | Test performance | Helped |
| Use Pre-PPO prompt selection (ablation) | PPO-based RLHF · Up to 2000 training steps | Test performance | No clear effect |
Surprising findings
- Scaling the newly collected prompt set from 1M to 6M did not improve RLHF performance and test scores peaked around step 3500 before declining.
- Increasing the proportion of Pre-PPO-selected newly collected data from 10% to 20% or 50% degraded performance.
Limitations
- Experiments use proprietary in-house models and internal evaluation sets, limiting reproducibility.
- Statistical significance is reported only for human evaluation; automatic gains are small (about +1 point).
- Edit distance as a granularity proxy is described as a coarse, computationally costly measure.
Experimental setup
Datasets: 1M original training prompts (public sources and human annotation) 5M newly collected open-source prompts
Benchmarks: internal TestSet V1.0 internal TestSet V2.0
Metrics: overall score across logical reasoning, IF, STEM, coding, NL human evaluation score p-value
Baselines: PPO-based RLHF baseline (original dataset) Initial-Run SFT Best-of-5
Builds on: PPO DPO pairRM Constitutional AI DeepSeek-R1
Closest papers by meaning
- Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.91)
- ODIN: Disentangled Reward Mitigates Hacking in RLHF (ICML 2024 · similarity 0.91)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.91)
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.91)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.91)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.