Research map / Large language models / Alignment and preferences
Self-Rewarding Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Rewarding Language Models that use the model's own LLM-as-a-Judge scores within an iterative DPO loop, improving both instruction following and reward modeling from a Llama 2 70B seed model.
Problem
Preference-based alignment is bottlenecked by human preference data and by frozen reward models that cannot improve during LLM training.
Approach
A single model both generates instruction-following responses and judges them via LLM-as-a-Judge prompting; the self-generated preference pairs are used for iterative DPO, so the reward-modeling ability improves alongside instruction following.
Key results
- AlpacaEval 2.0 win rate over GPT-4 Turbo rises from 9.94% (M1) to 15.38% (M2) to 20.44% (M3), outperforming Claude 2 (17.19%) and Gemini Pro (16.85%).
- Reward modeling pairwise accuracy on Open Assistant rises from 78.7% (M1) to 80.4% (M2) to 81.7% (M3), vs 65.1% for the SFT baseline.
- MT-Bench overall score improves from 6.78 (M1) to 7.25 (M3), though math/reasoning gains are smaller.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| iterate self-rewarding DPO training from M1 to M3 | Llama 2 70B Self-Rewarding · AlpacaEval 2.0 | win rate: 9.94% -> 20.44% | Helped |
| iterate self-rewarding DPO training from M1 to M3 | Llama 2 70B Self-Rewarding · MT-Bench | overall score: 6.78 ... 7.25 | Helped |
| train with AI Feedback Training preference pairs from the previous iteration | Self-Rewarding M1 · Open Assistant reward evaluation | pairwise accuracy: 78.7% -> 80.4% | Helped |
| continue iterative self-reward training from M2 to M3 | Self-Rewarding M2 · Open Assistant reward evaluation | pairwise accuracy: 80.4% -> 81.7% | Helped |
| add Evaluation Fine-Tuning (EFT) data (ablation) | IFT-only SFT Baseline · Open Assistant reward evaluation | pairwise accuracy: 65.1% -> 78.7% | Helped |
| add EFT data to seed training (ablation) | IFT-only instruction-following model · IFT test set | head-to-head win rate: 30.9% vs. 30.5% | No clear effect |
| replace the additive score-counting judge prompt with a multiple-choice prompt (ablation) | Llama 2 70B SFT Baseline · Open Assistant reward evaluation | pairwise accuracy: 65.1% -> 26.6% | Hurt |
| add only self-scored perfect-score examples instead of preference pairs (ablation) | SFT Baseline · IFT test set | head-to-head win rate: 29% wins vs 30% wins | No clear effect |
| use IFT+EFT seed training rather than IFT-only seed training (ablation) | Self-Rewarding training initialization · IFT test set | head-to-head win rate: 30.5% wins vs. 30.9% wins | No clear effect |
| fine-tune with successive self-generated preference data | Llama 2 70B · GSM8K | accuracy (em): 56.80 -> 60.27 -> 59.29 -> 57.70 | Mixed |
Surprising findings
- Adding only positive (score-5) self-generated examples to SFT gave no improvement over the SFT baseline, while preference pairs via DPO helped.
- Models trained with IFT data only (no EFT) often failed to follow the scoring format and their scores converged to 4, yielding very few valid training pairs.
Limitations
- Only three iterations run in a single setting, so scaling behavior is unknown.
- Response length increased across iterations (1092 to 2552 chars on AlpacaEval), a possible confound; reward hacking not analyzed.
- Math and reasoning gains are small, and NLP benchmark performance is roughly maintained or drops on some tasks; safety is not evaluated.
Experimental setup
Datasets: Open Assistant AlpacaEval MT-Bench ARC HellaSwag GSM8K MMLU NQ
Benchmarks: AlpacaEval 2.0 MT-Bench ARC HellaSwag GSM8K MMLU NQ Open Assistant held-out ranking set
Metrics: win rate pairwise accuracy exact match Spearman correlation Kendall tau MT-Bench score
Baselines: SFT Baseline Claude 2 Gemini Pro GPT-4 0613 GPT-4 0314 Mistral Medium PCO SPIN
Builds on: DPO Iterative DPO Self-Instruct Pairwise Cringe Optimization Constitutional AI
Closest papers by meaning
- SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.90)
- Learning Reward for Robot Skills Using Large Language Models via Self-Alignment (ICML 2024 · similarity 0.89)
- SALMON: Self-Alignment with Instructable Reward Models (ICLR 2024 · similarity 0.88)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.88)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.88)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.