Research map / Large language models / Alignment and preferences
RLCD: Reinforcement Learning from Contrastive Distillation for LM Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLCD, which builds RLHF preference pairs from outputs generated under contrasting positive and negative prompts (labeled by construction) instead of i.i.d. sampling plus AI scoring, then trains a preference model and runs PPO without human labels.
Problem
Collecting human pairwise preference data for RLHF is expensive; existing RLAIF produces i.i.d. output pairs that are often similar in quality, yielding noisy AI preference labels, while context distillation discards pairwise contrast.
Approach
Generate preference pairs from two contrasting prompts (positive prompt p+ encouraging the attribute, negative prompt p- encouraging its opposite), label o+ as preferred by construction, then train a preference model and run PPO, so outputs differ more and labels are cleaner.
Key results
- At 7B simulation scale, RLCD beats RLAIF on GPT-4 harmlessness win rate 84.8% vs 15.2%.
- RLCD preference model reaches 52.4% harmlessness accuracy vs 35.6% for RLAIF-7B, with RLAIF-30B below chance at 45.7%.
- At 30B simulation scale, GPT-4 prefers RLAIF over RLCD on some tasks (e.g., outlining 35.9 vs 64.1), while humans rate RLCD equal or better.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate preference pairs from contrasting positive and negative prompts, labeling the positive-prompt output preferred | LLaMA-7B alignment vs RLAIF-7B preference simulation · Helpfulness prompts | GPT-4 preferred (%): 90.7 / 9.3 | Helped |
| Generate preference pairs from contrasting positive and negative prompts, labeling the positive-prompt output preferred | LLaMA-7B alignment vs Context-Dist-7B preference simulation · Outlining prompts | GPT-4 preferred (%): 71.8 / 28.2 | Helped |
| Generate preference pairs from contrasting positive and negative prompts, labeling the positive-prompt output preferred | LLaMA-7B alignment vs RLAIF-30B preference simulation · Helpfulness prompts | GPT-4 preferred (%): 47.8 / 52.2 | Hurt |
| Generate preference pairs from contrasting positive and negative prompts, labeling the positive-prompt output preferred | LLaMA-7B alignment vs RLAIF-30B preference simulation · Outlining prompts | GPT-4 preferred (%): 35.9 / 64.1 | Hurt |
| Use prompt-implied preference labels instead of post-hoc rescoring (ablation) | RLCD-7B · Harmlessness prompts | GPT-4 preferred (%): 86.0 / 14.0 | Helped |
| Use prompt-implied preference labels instead of post-hoc rescoring (ablation) | RLCD-7B · Outlining prompts | GPT-4 preferred (%): 88.8 / 11.2 | Helped |
| Use prompt-implied preference labels instead of post-hoc rescoring (ablation) | RLCD-30B · Helpfulness prompts | GPT-4 preferred (%): 47.3 / 52.7 | Hurt |
| Use prompt-implied preference labels instead of post-hoc rescoring (ablation) | RLCD-30B · Outlining prompts | GPT-4 preferred (%): 36.4 / 63.6 | Hurt |
| Binarize RLAIF preference labels (ablation) | RLAIF-7B compared with RLCD-7B · Helpfulness prompts | GPT-4 preferred (%): 87.5 / 12.5 | Hurt |
| Use prompts focused purely on harmlessness (ablation) | RLCD-7B vs RLAIF-7B · Focused harmlessness prompts | GPT-4 preferred (%): 80.3 / 19.7 | Helped |
Surprising findings
- RLAIF harmlessness preference models with 30B scoring LLM perform worse than chance (45.7% accuracy) against human labels, yet still mitigate harm downstream.
- Rescoring RLCD outputs with RLAIF prompts is far worse than labeling by construction at 7B scale but becomes competitive at 30B scale.
Limitations
- Only aligns LLaMA-7B; not verified on larger pretrained models.
- Results depend on prompt design for preference generation and scoring.
- RLCD can produce irrelevant outputs when maximizing harmlessness, and outputs may be noisy.
Experimental setup
Datasets: Anthropic HH-RLHF harmlessness prompts Anthropic HH-RLHF helpfulness prompts 40000 internet story premises for outlining
Benchmarks: Anthropic HH-RLHF test prompts (harmlessness, helpfulness) story outlining prompts
Metrics: human Likert preference (normalized 1-8) GPT-4 pairwise win rate preference model binary accuracy vs. human labels preference model average probability
Baselines: LLaMA-7B (unaligned) RLAIF Context-Dist RLCD-Rescore RLAIF-Binary RLAIF-Few
Builds on: RLAIF Constitutional AI context distillation PPO AlpacaFarm
Closest papers by meaning
- Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR 2024 · similarity 0.89)
- Advantage-Guided Distillation for Preference Alignment in Small Language Models (ICLR 2025 · similarity 0.89)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.89)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.88)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.88)
- SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.