Research map / Reinforcement learning / RL from feedback
DPO Meets PPO: Reinforced Token Optimization for RLHF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RTO, which models RLHF as a token-level MDP, derives token-wise rewards from DPO's implicit reward, and optimizes them with PPO, outperforming PPO and direct preference baselines on AlpacaEval 2 and Arena-Hard with Llama-3-8B.
Problem
Standard PPO-based RLHF assigns a sparse sentence-level learned reward to only the last token, which is a mismatch with PPO's multi-step MDP design and makes learning sample-inefficient and sub-optimal.
Approach
Formulate RLHF as a token-level MDP and use the DPO-implicit reward (beta log ratio of DPO policy to reference) as a dense token-wise reward, combined with a KL term on the current policy and a small 1B sentence-level reward, then optimize with PPO.
Key results
- RTO reaches AlpacaEval 2 LC 27.00 vs PPO 19.47 (about +7.5 points) and Arena-Hard SC 20.3 vs PPO 16.2 (+4.1 points).
- RTO matches PPO performance with roughly 1/8 of the data and keeps improving with more data, while PPO saturates.
- Ablation: RS-PPO (DPO reward used only as shaping) recovers much of the gain, suggesting reward shaping is the main driver.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add token-wise DPO-derived rewards to PPO training | PPO · AlpacaEval 2 (LC) | win rate: 7.53% higher win rate | Helped |
| add token-wise DPO-derived rewards to PPO training | PPO · Arena-Hard (SC) | win rate: 4.1% higher win rate | Helped |
| assign rewards to each token instead of moving them to sentence delimiters or EoS (ablation) | RTO · AlpacaEval 2 and Arena-Hard | benchmark performance | Helped |
| use token-wise DPO reward for reward shaping while keeping total reward equivalent to the sentence-level reward (ablation) | PPO reward training (RS-PPO) · AlpacaEval 2 and Arena-Hard | benchmark performance | Helped |
| train with about 1/8 of the full dataset | RTO · AlpacaEval 2 and Arena-Hard | benchmark performance: 1/8 | No clear effect |
| increase training data | RTO versus PPO · AlpacaEval 2 and Arena-Hard | benchmark performance | Mixed |
Surprising findings
- The DPO implicit reward, although derived from sentence-level preferences, can serve as a useful token-level reward signal for PPO.
- Most of RTO's gain appears attributable to reward shaping rather than the exact magnitude of the DPO reward.
Limitations
- Theoretical guarantees assume linear reward and known deterministic transitions; the practical method uses DPO as a proxy for the optimal policy.
- Evaluation relies on AlpacaEval 2 and Arena-Hard with an LLM judge, which may carry verbosity/style biases.
Experimental setup
Datasets: UltraFeedback (binarized) ultra_train prompt-only split
Benchmarks: AlpacaEval 2 Arena-Hard
Metrics: length-controlled win rate win rate style-controlled win rate
Baselines: PPO DPO R-DPO SimPO TDPO SFT
Builds on: PPO DPO KL-regularized RLHF Bradley-Terry model
Closest papers by meaning
- TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization (ICML 2025 · similarity 0.94)
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (ICML 2024 · similarity 0.93)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.91)
- Policy Filtration for RLHF to Mitigate Noise in Reward Models (ICML 2025 · similarity 0.91)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
- Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.