Research map / Large language models / Instruction and fine-tuning
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyses how SFT, best-of-N, and RLHF (PPO) affect OOD generalisation and output diversity for LLaMa-7B on summarisation and instruction following, finding RLHF generalises better but yields substantially less output diversity than SFT.
Problem
Understanding how each stage of the RLHF pipeline (SFT, reward modelling, RL) affects out-of-distribution generalisation and output diversity of LLMs, which has been underexplored.
Approach
Systematically compares SFT, best-of-N, and RLHF policies on ID vs OOD test sets (TL;DR to CNN/DailyMail; AlpacaEval and Sequential Instructions) using GPT-4 simulated preferences and multiple diversity metrics (EAD, Sentence-BERT, NLI) measured per-input and across-input.
Key results
- On summarisation, RLHF's per-input diversity is substantially lower than SFT on EAD and Sent BERT, while NLI shows no meaningful difference.
- On the harder Sequential Instructions OOD set, RLHF generalises much better than SFT, while on AlpacaEval all models generalise similarly.
- Best-of-16 outperforms RLHF which outperforms SFT on summarisation ID and OOD; GPT-4 judge agrees with humans at 69-74% per-example agreement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RLHF instead of SFT | LLaMa 7B summarisation · TL;DR and CNN/DailyMail | preference vs reference (PvR) | Helped |
| Use Bo16 instead of SFT | LLaMa 7B summarisation · TL;DR and CNN/DailyMail | preference vs reference (PvR) | Helped |
| Use RLHF instead of SFT | LLaMa 7B instruction following · AlpacaFarm Self-Instruct, AlpacaEval, and Sequential Instructions | preference vs reference (PvR) | Helped |
| Use RLHF instead of SFT | LLaMa 7B instruction following · Sequential Instructions | generalisation gap | Helped |
| Use RLHF instead of SFT | LLaMa 7B instruction following · AlpacaEval | head-to-head winrate change from ID to OOD: approximately 3.5% | Helped |
| Use Bo16 instead of SFT | LLaMa 7B instruction following · AlpacaEval | head-to-head winrate change from ID to OOD: approximately 3.5% | Helped |
| Use RLHF instead of SFT | LLaMa 7B summarisation · Per-input output diversity | EAD and Sentence-BERT diversity | Hurt |
| Use RLHF instead of SFT | LLaMa 7B summarisation · Across-input output diversity | EAD and Sentence-BERT diversity | Hurt |
| Use RLHF instead of SFT | LLaMa 7B summarisation · Per-input and across-input output diversity | NLI diversity | No clear effect |
| Increase the KL penalty coefficient (ablation) | RLHF summarisation policy · TL;DR and CNN/DailyMail | GPT-4 preference vs reference | Hurt |
| Increase the KL penalty coefficient (ablation) | RLHF summarisation policy · Summarisation per-input diversity | per-input diversity | Hurt |
| Use Bo16 temperature 1 instead of temperature 0.7 (ablation) | Bo16 summarisation · TL;DR and CNN/DailyMail | GPT-4 win rate vs reference | Hurt |
Surprising findings
- Across-input diversity gap between RLHF and SFT is much smaller than per-input gap, suggesting across-input mode collapse.
- Increasing the KL penalty coefficient reduced per-input diversity rather than increasing it.
Limitations
- No theoretical explanation for the observed effects.
- Only SFT, RLHF and BoN considered; other fine-tuning methods not evaluated.
- Relies on automatic GPT-4 evaluation and metrics without broad human evaluation; instruction-following diversity not meaningfully measured due to long outputs.
Experimental setup
Datasets: TL;DR CNN/DailyMail AlpacaFarm Self-Instruct AlpacaEval
Benchmarks: TL;DR CNN/DailyMail AlpacaEval Sequential Instructions AlpacaFarm Self-Instruct
Metrics: GPT-4 win rate vs reference generalisation gap head-to-head win rate expectation-adjusted distinct n-grams Sentence-BERT cosine similarity NLI diversity
Baselines: SFT Best-of-N (N=16)
Builds on: PPO Stiennon et al. 2020 RLHF summarisation AlpacaFarm Self-Instruct
Closest papers by meaning
- KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity (NeurIPS 2025 · similarity 0.91)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.91)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.91)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.90)
- Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.90)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.