Research map / Large language models / Alignment and preferences
SELF-EVOLVED REWARD LEARNING FOR LLMS
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Evolved Reward Learning (SER), where a reward model iteratively self-labels unlabeled preference data, filters high-confidence pairs via learning-status checks, and retrains, starting from only 15% of human labels, before PPO on the evolved RM.
Problem
Training reliable reward models for RLHF requires costly high-quality human or strong-AI preference labels, which limits scalability and may not keep helping as LLMs improve.
Approach
The reward model labels unlabeled preference pairs itself, and a learning-status test (Status 1: confident good/bad separation; Status 2: fine-grained score differences between paired answers) chooses which self-labeled pairs to add for pairwise-loss retraining, iterating until convergence before PPO with the evolved RM.
Key results
- SER improves reward modeling accuracy over Loop 0 across settings, e.g., HH-RLHF Llama3-8b from 56.9 to 68.56 and UltraFeedback Llama3-8b from 62.54 to 74.46.
- Reported average 7.88% accuracy gain over the 15%-data seed model; on Summarize Mistral-7b SER reaches 65.49 vs full-data 63.56 (reported as +1.93%).
- On StackOverflow Llama3-8b SER reaches 70.8 vs full-data 69, surpassing full-data training by about 1.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Iteratively self-label and retrain the reward model using selected data | Reward model initialized with 15% human-labeled data · HH-RLHF | accuracy: 56.9 -> 68.56 | Helped |
| Iteratively self-label and retrain the reward model using selected data | Reward model initialized with 15% human-labeled data · Summarize | accuracy: 58.01 -> 68.42 | Helped |
| Iteratively self-label and retrain the reward model using selected data | Reward model initialized with 15% human-labeled data · UltraFeedback | accuracy: 62.54 -> 74.46 | Helped |
| Iteratively self-label and retrain the reward model using selected data | Reward model initialized with 15% human-labeled data · StackOverflow | accuracy: 69 -> 70.8 | Helped |
| Train on hard, ambiguous samples in Loop 3 using learning status 2 (ablation) | SER reward model · HH-RLHF, UltraFeedback, and Summarize | accuracy | Helped |
| Continue training on similar, simple samples in Loop 2 (ablation) | SER reward model · UltraFeedback (Llama 13B) | accuracy | Hurt |
| Use self-evolved RM instead of the full-data RM to guide PPO | LLM trained with PPO · Anthropic HH-RLHF | win rate | Helped |
| Use self-evolved RM instead of the full-data RM to guide PPO | LLM trained with PPO · StackOverflow | win rate | Helped |
Surprising findings
- Additional self-labeled easy data in later loops can degrade performance (e.g., Llama 13B on UltraFeedback), motivating the switch to hard-pair filtering.
- Self-evolved RM sometimes exceeds the full-human-label RM despite using only 15% of human labels.
Limitations
- Theoretical convergence analysis relies on strong assumptions (e.g., accuracy >0.5 on high-confidence data) and the PPO bound is loose/heuristic.
- Thresholds (tau_high, tau_low, tau_delta) and margins are tuned empirically, with no robust automatic identification of learning status.
- Full iterative loop with LLM generation (step 4 in each iteration) was not tested due to PPO compute cost.
Experimental setup
Datasets: HH-RLHF UltraFeedback Summarize (TL;DR) StackOverflow
Benchmarks: HH-RLHF UltraFeedback Summarize StackOverflow
Metrics: reward model pairwise accuracy win rate
Baselines: Loop 0 reward model (15% human data) full-dataset reward model SFT baseline
Builds on: RLHF PPO DPO Self-Rewarding Language Models
Closest papers by meaning
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.93)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.92)
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.91)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.90)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.90)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.