Research map / Large language models / Alignment and preferences
Token-level Direct Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TDPO, a token-level extension of DPO that adds per-token forward KL constraints via a token-level Bradley-Terry reformulation, evaluated on IMDb, Anthropic HH and MT-Bench.
Problem
DPO enforces KL divergence only at the sentence level with a reverse KL constraint, so divergence from the reference model grows unevenly between preferred and dispreferred responses, hurting diversity and divergence efficiency.
Approach
Reformulates the Bradley-Terry preference model at the token level using advantage functions, yielding a DPO-style loss with an added sequential forward-KL term per token; TDPO2 stop-gradients the preferred-response KL term and scales the dispreferred term by alpha.
Key results
- On Anthropic HH, TDPO2 reaches 67.33% alignment accuracy and 4.915 entropy, versus DPO's 59.43% and 3.196.
- On IMDb, TDPO1 and TDPO2 achieve higher reward than DPO at matched forward KL on the frontier.
- On MT-Bench, TDPO2 achieves a higher GPT-4-judged win rate than DPO, TDPO1 and PPO-based RLHF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TDPO2 instead of DPO | DPO · Anthropic HH test set | accuracy (%): 59.43 -> 67.33 | Helped |
| Use TDPO2 instead of DPO | DPO · Anthropic HH test set | entropy: 3.196 -> 4.915 | Helped |
| Use TDPO2 instead of f-DPO (FKL) | f-DPO (FKL) · Anthropic HH test set | accuracy (%): 54.71 -> 67.33 | Helped |
| Use TDPO2 instead of f-DPO (FKL) | f-DPO (FKL) · Anthropic HH test set | entropy: 4.708 -> 4.915 | Helped |
| Add the TDPO2 stop-gradient treatment and alpha-weighted KL term (ablation) | TDPO1 · Anthropic HH test set | accuracy (%): 60.08 -> 67.33 | Helped |
| Add the TDPO2 stop-gradient treatment and alpha-weighted KL term (ablation) | TDPO1 · Anthropic HH test set | entropy: 4.727 -> 4.915 | Helped |
| Increase alpha to higher values (ablation) | TDPO2 · IMDb sentiment generation | reward optimization | Hurt |
| Use TDPO1 instead of TDPO2 (ablation) | TDPO2 · IMDb sentiment generation | sequential KL divergence control | Hurt |
Surprising findings
- Higher alpha values in TDPO2 made reward optimization harder, so alpha=0.5 was used for the remaining experiments.
Limitations
- Hyperparameter alpha trades off KL control against optimization speed and difficulty.
- Evaluation relies heavily on GPT-4 judgments and automatic metrics, with small-scale models (GPT-2 Large, Pythia-2.8B).
Experimental setup
Datasets: IMDb Anthropic HH
Benchmarks: IMDb reward-KL frontier Anthropic HH MT-Bench
Metrics: average reward vs sequential KL frontier alignment accuracy predictive entropy (diversity) win rate win/tie/lose rate
Baselines: DPO f-DPO (forward KL) PPO SFT
Builds on: DPO f-DPO TRPO Regret Preference Model RLHF PPO
Closest papers by meaning
- KL Penalty Control via Perturbation for Direct Preference Optimization (NeurIPS 2025 · similarity 0.91)
- TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights (ICLR 2025 · similarity 0.91)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.91)
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.90)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.90)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.