Research map / Large language models / Alignment and preferences
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares RLAIF (LLM-labeled preferences distilled into a reward model) with RLHF on summarization and dialogue tasks, and introduces d-RLAIF, which uses an off-the-shelf LLM's scores directly as RL reward.
Problem
Human preference labels for RLHF are expensive and hard to scale; it was unknown whether LLM-generated preference labels (RLAIF) can match human feedback for aligning LLM policies.
Approach
Label preferences with an off-the-shelf PaLM 2 LLM using scored log-probabilities of "1"/"2" with position debiasing and optional chain-of-thought, train a reward model on those soft labels (canonical RLAIF), or skip RM training and let the LLM score each generation 1-10 directly as the RL reward (d-RLAIF).
Key results
- RLAIF vs SFT win rate 71% and RLHF vs SFT 73% on summarization; RLAIF vs RLHF 50% (no significant difference).
- On harmless dialogue, RLAIF harmless rate 88% vs RLHF 76% and SFT 64%.
- d-RLAIF vs SFT 74% on summarization, and d-RLAIF beats same-size RLAIF 60% head-to-head.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace human preference labels with AI preference labels for reward-model training | RLAIF vs. RLHF · Reddit TL;DR and helpful dialogue | human win rate vs. SFT: 71% vs 73%; 63% vs 64% | No clear effect |
| Replace human preference labels with AI preference labels for reward-model training | Harmless dialogue policy · Harmless dialogue test | harmless rate: 76% -> 88% | Helped |
| Use PaLM 2 XS as the AI labeler instead of a larger model | Same-size RLAIF · Reddit TL;DR test | win rate vs. SFT: 68% | Helped |
| Use direct LLM rewards during RL instead of distilling AI preferences into a reward model | d-RLAIF · Reddit TL;DR test | win rate vs. Same-size RLAIF: 60% | Helped |
| Use direct LLM rewards during RL | d-RLAIF · Helpful dialogue test | win rate vs. SFT: 66% | Helped |
| Add chain-of-thought reasoning to the base zero-shot prompt (ablation) | AI preference labeler · Summarization, helpful dialogue, and harmless dialogue | AI labeler alignment: 76.1% -> 77.5%; 67.8% -> 69.1%; 69.4% -> 70.6% | Helped |
| Use a detailed preamble instead of a base preamble (ablation) | AI preference labeler · Summarization | AI labeler alignment: 76.1% -> 77.4% | Helped |
| Use a detailed preamble instead of a base preamble (ablation) | AI preference labeler · Helpful and harmless dialogue | AI labeler alignment: 67.8% -> 67.6%; 69.4% -> 70.1% | Mixed |
| Add in-context exemplars (ablation) | AI preference labeler · Summarization and helpful dialogue | AI labeler alignment: 76.1% -> 76.0% -> 75.7%; 67.8% -> 67.1% -> 66.8% | Hurt |
| Add in-context exemplars (ablation) | AI preference labeler · Harmless dialogue | AI labeler alignment: 69.4% -> 71.7% -> 72.1% | Helped |
| Increase the AI labeler size (ablation) | AI preference labeler · Summarization preference holdout | AI labeler alignment: 62.7% -> 73.8% -> 78.0% | Helped |
| Combine human and AI feedback for reward-model training (ablation) | RLHF + RLAIF · Reddit TL;DR test | win rate vs. SFT: 74% -> 71% | Hurt |
Surprising findings
- In-context learning (few-shot exemplars) decreased AI labeler alignment for summarization and helpfulness as shots increased.
- Self-consistency with sampled chain-of-thought strictly degraded AI labeler alignment.
Limitations
- Only one model family (PaLM 2) and a single AI labeler family were studied; results may not transfer.
- Length bias: RLAIF and RLHF responses were longer; length-correction was post-hoc and imperfect.
- Combining RLHF and RLAIF feedback did not improve over RLHF alone.
Experimental setup
Datasets: Reddit TL;DR OpenAI Human Preferences Anthropic Helpful and Harmless human preferences
Benchmarks: Reddit TL;DR Anthropic HH helpful test Anthropic HH harmless test
Metrics: win rate harmless rate AI labeler alignment pairwise reward model accuracy length-controlled win rate
Baselines: RLHF SFT same-size RLAIF canonical RLAIF
Builds on: RLHF Constitutional AI REINFORCE Reddit TL;DR
Closest papers by meaning
- SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.93)
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.92)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.92)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.92)
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.92)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.