Research map / Reinforcement learning / RL from feedback
Learn Your Reference Model for Real Good Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Trust Region variants of offline alignment (TR-DPO, TR-IPO, TR-KTO) that periodically or softly update the reference policy to mitigate overoptimization, showing gains on Anthropic-HH, Reddit TL;DR, AlpacaEval 2 and Arena-Hard.
Problem
Offline LLM alignment methods (DPO, IPO, KTO) suffer from overoptimization as the trained policy drifts far from a fixed SFT reference, degrading sample quality, likely through probability leakage onto out-of-domain sequences.
Approach
Dynamically update the reference policy during training, either softly (exponential moving merge of the policy into the reference at every step) or hardly (replace the reference with the current policy every tau steps), so the implicit KL trust region moves with training.
Key results
- On Llama3-Base 8B, TR-DPO^tau reaches 27.7 LC win rate on AlpacaEval 2 vs 18.2 for DPO; TR-IPO^alpha reaches 29.5 vs 14.4 for IPO.
- On Pythia 2.8B Anthropic-HH with AutoSxS, TR-DPO^alpha (alpha=0.6) wins 42.4% vs 32.4% losses against DPO.
- TR variants with hard update improve over base methods on Llama3-Instruct AlpacaEval 2 (e.g., TR-KTO^tau 41.3 vs KTO 33.1 LC).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TR-DPO with hard reference updates instead of DPO | Llama3-Base (8B) · AlpacaEval 2 | LC%: 18.2 -> 27.7 | Helped |
| Use TR-IPO with soft reference updates instead of IPO | Llama3-Base (8B) · AlpacaEval 2 | LC%: 14.4 -> 29.5 | Helped |
| Use TR-KTO with hard reference updates instead of KTO | Llama3-Base (8B) · AlpacaEval 2 | LC%: 14.2 -> 16.5 | Helped |
| Use TR-DPO with soft reference updates instead of DPO | Llama3-Base (8B) · Arena-Hard | WR%: 15.9 -> 21.3 | Helped |
| Use TR-IPO with hard reference updates instead of IPO | Llama3-Base (8B) · Arena-Hard | WR%: 17.8 -> 21.1 | Helped |
| Use TR-KTO with soft reference updates instead of KTO | Llama3-Base (8B) · Arena-Hard | WR%: 12.5 -> 14.9 | Helped |
| Use soft reference updates with α = 0.8 instead of baseline KTO | Llama3-Base (8B) · AlpacaEval 2 | LC%: 14.2 -> 14.7 | Mixed |
| Vary the soft-update weight α (ablation) | TR-DPO, TR-IPO, and TR-KTO · Anthropic-HH, Pythia 2.8B | Win | Mixed |
| Vary the hard-update interval τ (ablation) | TR-DPO, TR-IPO, and TR-KTO · Anthropic-HH, Pythia 2.8B | Win | Mixed |
| Set α = 0.9 (ablation) | TR methods · Training and generation | stability | Hurt |
| Use τ values smaller than 16 (ablation) | TR methods · Training and generation | stability | Hurt |
Surprising findings
- Higher-KL TR policies achieve better human-centric metrics, implying that moving further from the SFT reference can be beneficial rather than harmful.
- Keeping probability mass on chosen/rejected texts matters less than preventing probability leakage to OOD sequences for final quality.
Limitations
- Large tau values in hard updates may not scale well to smaller datasets; hyperparameters alpha and tau are sensitive to task and model size.
- Evaluation relies on GPT-4 automatic judges rather than human evaluation.
Experimental setup
Datasets: Anthropic-HH Reddit TL;DR summarization UltraChat-200k UltraFeedback
Benchmarks: Anthropic-HH Reddit TL;DR AlpacaEval 2 Arena-Hard MixEval
Metrics: win rate length-controlled win rate tie rate KL divergence probability of improvement Self-BLEU
Baselines: DPO IPO KTO SFT
Builds on: DPO IPO KTO Trust Region optimization
Closest papers by meaning
- Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models (NeurIPS 2025 · similarity 0.90)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.89)
- Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples (ICML 2025 · similarity 0.89)
- Your Weak LLM is Secretly a Strong Teacher for Alignment (ICLR 2025 · similarity 0.88)
- Improving Model Alignment Through Collective Intelligence of Open-Source Models (ICML 2025 · similarity 0.88)
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.