Research map / Large language models / Alignment and preferences
Bootstrapping Language Models with DPO Implicit Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Bootstraps a DPO-tuned LLM by using its own DPO implicit rewards to label on-policy samples for further DPO rounds, with length-regularized reward shaping and experience replay (DICE).
Problem
Iterative self-alignment of a DPO-tuned LLM without external feedback, where naive implicit-reward labeling introduces length exploitation and over-reliance on an imperfect implicit reward can erode initial knowledge.
Approach
Use the implicit reward of the current DPO policy (relative to the previous-round policy) to rank K on-policy samples per prompt, add a length-regularized reward shaping term whose strength is searched so the winner/loser length difference is near zero, and mix in offline preference data via experience replay before the next DPO round.
Key results
- DICE improves LC win rate on AlpacaEval 2 by 8.02 points for Zephyr-7B-beta (12.69 to 20.71) and 9.35 points for Llama-3-8B-DPO (18.20 to 27.55) after two iterations.
- DICE Llama3 8B Iter 2 (27.55 LC) outperforms Gemini Pro (24.38 LC) and Llama 3 8B Instruct (22.92 LC) on the AlpacaEval 2 leaderboard table.
- Offline DPO over two iterations degrades Zephyr to 4.96 LC, well below the base 12.69.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| bootstrap DPO using implicit-reward preferences, length-regularized reward shaping, and replay | Zephyr-7B-beta · AlpacaEval 2 | LC win rate: 12.69 -> 20.71 | Helped |
| bootstrap DPO using implicit-reward preferences, length-regularized reward shaping, and replay | Llama-3-8B-DPO · AlpacaEval 2 | LC win rate: 18.20 -> 27.55 | Helped |
| continue offline DPO training on the fixed dataset | Zephyr-7B-beta · AlpacaEval 2 | LC win rate: 13.40 -> 4.96 | Hurt |
| remove length-regularized reward shaping (α=0.000 instead of α*=0.023) (ablation) | DICE, Zephyr-7B-beta, γ=0.5 · AlpacaEval 2 | LC win rate: 19.03 -> 15.92 | Hurt |
| increase length penalty from α*=0.023 to 2α*=0.046 (ablation) | DICE, Zephyr-7B-beta, γ=0.5 · AlpacaEval 2 | LC win rate: 19.03 -> 14.91 | Hurt |
| use α*=0.023 rather than no reward shaping (ablation) | DICE, Zephyr-7B-beta, γ=0.0 · AlpacaEval 2 | LC win rate: 13.32 -> 18.88 | Helped |
| increase the offline replay ratio to γ=0.5 from γ=0.0 (ablation) | DICE, Zephyr-7B-beta, α=0.023 · AlpacaEval 2 | LC win rate: 18.88 -> 19.03 | Mixed |
| choose replay ratio γ=0.5 instead of γ=0.25 (ablation) | DICE, Zephyr-7B-beta · Arena-Hard | LC score: 16.1 -> 18.4 | No clear effect |
| replace offline-only IPO training with DICE-generated data plus IPO | IPO, Zephyr-7B-beta · AlpacaEval 2 | LC win rate: 14.83 -> 18.51 | Helped |
| replace offline-only KTO training with DICE-generated data plus KTO | KTO, Zephyr-7B-beta · AlpacaEval 2 | LC win rate: 13.92 -> 14.88 | Helped |
| use DPO implicit rewards instead of an offline-trained internal scalar reward model | self-alignment reward labeling · GPT-4o preference-label alignment | alignment rate: 0.624 -> 0.698 | Helped |
| increase DICE from Iter 2 to Iter 3 (ablation) | DICE, Zephyr-7B-beta · AlpacaEval 2 | LC win rate: 20.71 -> 14.02 | Hurt |
Surprising findings
- Continuing offline DPO on a fixed dataset for multiple rounds performs worse than the base model.
- Performance drops at iteration 3 for both DICE (LC 14.02 on Zephyr) and the LLM-as-a-Judge baseline, so gains do not continue beyond two self-alignment rounds.
Limitations
- Depends on a well-trained initial DPO model; a poorly trained implicit reward can collapse the pipeline.
- No continued improvement beyond about two to three iterations; iteration-3 performance drops.
- Relies on a fixed prompt set across iterations and an extra hyperparameter gamma that must be tuned.
Experimental setup
Datasets: UltraFeedback AlpacaFarm prompts
Benchmarks: AlpacaEval 2 Arena-Hard
Metrics: length-controlled win rate win rate alignment rate with GPT-4o labels
Baselines: Offline DPO Offline DPO w/ new ref LLM-as-a-Judge Self-rewarding LM Length-regularized DPO PPO IPO KTO Hinge loss ERM-555k IntlRM
Builds on: DPO iterative DPO Zephyr length-regularized DPO experience replay
Closest papers by meaning
- AlphaPO: Reward Shape Matters for LLM Alignment (ICML 2025 · similarity 0.93)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.92)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.91)
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.