Research map / Reinforcement learning / RL from feedback
All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning
Gokul Swamy, Sanjiban Choudhury, Wen Sun, Steven Wu, Drew Bagnell
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use online DPO (RM-labeled on-policy samples) instead of offline DPO | Pythia on tl;dr summarization · tl;dr (gpt-4o winrate) | winrate vs human reference | Improved |
| run online DPO on top of the offline DPO policy using an RM trained on the same preference data | offline DPO policy · tl;dr (gpt-4o winrate) | winrate | Improved |
| train offline DPO for two epochs (DPO 2x) to match gradient steps (ablation) | offline DPO · tl;dr (gpt-4o winrate) | winrate | No clear effect |
| augment online DPO prompts with SFT-dataset prompts (about 3x data) (ablation) | online DPO · tl;dr (gpt-4o winrate) | winrate | No clear effect |
| restrict preference data to SFT-policy samples labeled by gpt-4o (narrower data) (ablation) | online DPO over offline DPO · tl;dr (narrow on-policy data) | winrate | Improved |
| scale reward model size to be much smaller or larger than the policy (ablation) | global RM for best-of-N · tl;dr BoN | BoN winrate | No clear effect |
| scale up the generation policy size (ablation) | best-of-N with fixed RM · tl;dr BoN | BoN winrate | Improved |
| add online DPO on two-word summarization (horizon about 1/10) (ablation) | offline DPO · two-word summarization (gpt-4o labels) | winrate: about +1% winrate | No clear effect |
| add online DPO with learned global RM when labels and eval use ROUGE-L (ablation) | offline DPO · tl;dr (ROUGE-L reward) | ROUGE-L | No clear effect |
| train offline DPO for one extra epoch (ablation) | offline DPO · tl;dr (ROUGE-L reward) | ROUGE-L | Improved |
| use a global RM instead of a token-wise local RM (ablation) | reward model · tl;dr validation likelihood | validation likelihood | Improved |
| add reference-policy regularization to the local RM (DPO RM vs Local) (ablation) | local RM · tl;dr validation likelihood | validation likelihood | Hurt |
Closest papers by meaning
- Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.89)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.89)
- The Importance of Online Data: Understanding Preference Fine-tuning via Coverage (NeurIPS 2024 · similarity 0.88)
- Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective (ICML 2025 · similarity 0.88)
- What Makes a Reward Model a Good Teacher? An Optimization Perspective (NeurIPS 2025 · similarity 0.88)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
