Research map / Reinforcement learning / RL from feedback
TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TGDPO, which decomposes sequence-level PPO into token-level PPO with token-level reward guidance and derives a DPO loss that reweights per-token log-ratios using the induced DPO token reward.
Problem
DPO is a sequence-level bandit objective, so it cannot directly incorporate fine-grained token-level reward guidance that has helped PPO; the partition function also complicates such integration.
Approach
Decompose sequence-level PPO into token-level PPO problems with a modified reward scaled by a function f of the token reward, derive the closed-form optimal token policy, show the partition-function term can be dropped from the Bradley-Terry likelihood, and set f(r) = 1 ± alpha*r so tokens deviate from the reference policy in proportion to their induced DPO reward.
Key results
- On Llama3-8B-Instruct (PairRM), TGDPO reaches 43.9% AlpacaEval 2 win rate vs 41.7% for DPO and 34.3% Arena-Hard vs 30.4% for DPO.
- With ArmoRM on Llama3.2-3B-Instruct, TGDPO improves MT-Bench win rate to 36.9% from 29.4% for DPO.
- On SFT Llama3-8B, TGDPO with SimPO's token reward gets 26.9% AlpacaEval 2 vs 16.4% for SimPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add token-level reward guidance to DPO | Llama3-8B-Instruct with PairRM preferences · AlpacaEval 2 | win rate (%): 41.7 -> 43.9 | Helped |
| Add token-level reward guidance to DPO | Llama3-8B-Instruct with PairRM preferences · Arena-Hard | win rate (%): 30.4 -> 34.3 | Helped |
| Add token-level reward guidance to DPO | Llama3-8B-Instruct with PairRM preferences · MT-Bench | win rate (%): 37.5 -> 41.9 | Helped |
| Add token-level reward guidance to DPO | Llama3-8B-Instruct with ArmoRM preferences · MT-Bench | score: 8.2 -> 7.9 | Hurt |
| Train to loss convergence (ablation) | TGDPO (Llama3-8B-Instruct PairRM) · AlpacaEval 2 | win rate (%): 41.7 -> 43.9 | Helped |
| Train to loss convergence (ablation) | TGDPO (Llama3-8B-Instruct PairRM) · Arena-Hard | win rate (%): 30.4 -> 34.3 | Helped |
| Set convergence speed to alpha = 1.0 instead of alpha = 0.5 (ablation) | TGDPO · AlpacaEval 2 | win rate (%): 43.9 -> 42.5 | Hurt |
| Set convergence speed to alpha = 2.0 instead of alpha = 0.5 (ablation) | TGDPO · Arena-Hard | win rate (%): 34.3 -> 34.3 | No clear effect |
| Use token rewards from DPO with beta = 0.1 instead of beta = 0.01 (ablation) | TGDPO · AlpacaEval 2 | win rate (%): 43.9 -> 42.8 | Hurt |
| Use token rewards from DPO with beta = 0.1 instead of beta = 0.01 (ablation) | TGDPO · Arena-Hard | win rate (%): 34.3 -> 34.3 | No clear effect |
| Use SimPO's token-level reward instead of DPO's token-level reward (ablation) | TGDPO from Llama3-8B-SFT-Mixture · AlpacaEval 2 | win rate (%): 12.8 -> 26.9 | Helped |
| Use SimPO's token-level reward instead of DPO's token-level reward (ablation) | TGDPO from Llama3-8B-SFT-Mixture · Arena-Hard | win rate (%): 13.8 -> 25.3 | Helped |
Surprising findings
- Under loss convergence, DPO and SimPO degrade substantially (SimPO falls to 4.6% AlpacaEval 2) while TGDPO stays strong.
- TGDPO performs nearly identically whether the token rewards come from a DPO model trained with beta=0.1 or beta=0.01.
Limitations
- Evaluation focuses on helpfulness; safety, honesty and fairness are not assessed.
- Theoretical elimination of the partition function relies on approximations and assumptions (e.g. alpha small enough, f positive).
Experimental setup
Datasets: UltraFeedback
Benchmarks: AlpacaEval 2 Arena-Hard MT-Bench
Metrics: win rate MT-Bench score
Baselines: DPO SimPO TDPO SFT
Builds on: DPO SimPO TDPO Bradley-Terry model PPO
Closest papers by meaning
- DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.94)
- SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024 · similarity 0.90)
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (ICML 2024 · similarity 0.90)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.90)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.89)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.