Research map / Reinforcement learning / RL from feedback
Unifying Stable Optimization and Reference Regularization in RLHF
Li He, Qiang Qu, He Zhao, Stephen Wan, Dadong Wang, Lina Yao, Tongliang Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dual-KL penalties (pi_0 and pi_t) to advantage (ablation) | PPO · Helpfulness | mean reward: 9.070 -> 9.522 | Improved |
| add dual-KL penalties with ratio clipping (ablation) | PPO · Helpfulness | mean reward: 9.070 -> 9.581 | Improved |
| add ratio clipping to Dual-PPO (ablation) | Dual-PPO · Helpfulness | win rate over pi_0: 88.40% -> 90.37% | Improved |
| replace PPO with DAR | PPO · TL;DR (Qwen2) | reference win rate: 72.87% -> 98.27% | Improved |
| replace GRPO with DAR | GRPO · Mean win rate (Qwen2) | reference win rate: 85.15% -> 92.42% | Improved |
| replace GRPO with DAR | GRPO · Harmlessness (Qwen2) | reference win rate: 85.50% -> 85.84% | No clear effect |
| replace RLOO with DAR | RLOO · Mean win rate (Qwen2.5) | reference win rate: 76.65% -> 82.60% | Improved |
| replace online DPO with DAR | online DPO · Mean win rate (Qwen2) | reference win rate: 83.63% -> 92.42% | Improved |
| use online data instead of offline preference data | DPO · Mean win rate (Qwen2) | reference win rate: 75.47% -> 83.63% | Improved |
| replace GRPO with DAR | GRPO · MT-Bench GPT-4 | score: 8.425 -> 8.538 | Improved |
| replace RLOO with DAR | RLOO · AlpacaEval 2.0 | LC win rate: 52.25 -> 54.17 | Improved |
| use RL-based dual-KL (DAO) instead of regression (DAR) (ablation) | DAR · Helpfulness and TL;DR reward curves | training stability | Hurt |
Closest papers by meaning
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.92)
- KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity (NeurIPS 2025 · similarity 0.91)
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.91)
- Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.90)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.90)
- Clone-Robust AI Alignment (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
