Research map / Large language models / Alignment and preferences
Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D²PO, a DPO variant that multiplies each token's log-ratio reward by an exponential temporal decay factor γ^t so earlier tokens contribute more, with a reference-free on-policy variant.
Problem
DPO suffers from length bias and treats all token rewards in a sequence uniformly, ignoring that earlier tokens may matter more for alignment.
Approach
Insert an exponential temporal decay γ^t into the summed sequence log-ratio rewards of the DPO loss so earlier tokens get larger weights; for on-policy data, drop the reference model (reference-free variant), which also reduces length bias.
Key results
- On-policy setup, D²PO beats DPO by 5.9-8.8 points on AlpacaEval 2 and 3.3-9.7 points on Arena-Hard across Llama3-8B, Gemma2-9B and Mistral-NeMo-12B (per abstract).
- Reference-free D²PO reaches 62.4 LC win rate on AlpacaEval 2 and 63.6 win rate on Arena-Hard with Mistral-NeMo-12B, outperforming SimPO on the same setup.
- On OpenLLM benchmarks, D²PO improves MATH for Llama3-8B to 8.5 from DPO's 7.8, and keeps MMLU near SFT levels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply temporal decay (γ=0.98) instead of DPO | Llama3-Instruct (8B) · AlpacaEval 2 LC win rate, on-policy | LC win rate (%): 47.60 -> 53.54 | Helped |
| Apply temporal decay (γ=0.98) instead of DPO | Llama3-Instruct (8B) · Arena-Hard win rate, on-policy | win rate (%): 42.4 -> 47.3 | Helped |
| Apply temporal decay (γ=0.98) instead of DPO | Gemma2-Instruct (9B) · AlpacaEval 2 LC win rate, on-policy | LC win rate (%): 53.43 -> 59.71 | Helped |
| Apply temporal decay (γ=0.98) instead of DPO | Gemma2-Instruct (9B) · Arena-Hard win rate, on-policy | win rate (%): 56.7 -> 66.4 | Helped |
| Apply temporal decay (γ=0.98) instead of DPO | Mistral-NeMo-Instruct (12B) · AlpacaEval 2 LC win rate, on-policy | LC win rate (%): 53.32 -> 62.07 | Helped |
| Apply temporal decay (γ=0.98) instead of DPO | Mistral-NeMo-Instruct (12B) · Arena-Hard win rate, on-policy | win rate (%): 59.0 -> 62.3 | Helped |
| Remove the reference model from D²PO | D²PO (γ=0.98), Gemma2-Instruct (9B) · AlpacaEval 2 LC win rate, on-policy | LC win rate (%): 58.42 -> 60.76 | Helped |
| Replace exponential decay with head decay (ablation) | D²PO, Gemma2-Instruct (9B) · AlpacaEval 2 LC win rate | LC win rate (%): 59.7 -> 54.7 | Hurt |
| Replace exponential decay with linear decay (ablation) | D²PO, Gemma2-Instruct (9B) · AlpacaEval 2 LC win rate | LC win rate (%): 59.7 -> 54.5 | Hurt |
| Replace exponential decay with power-law decay (ablation) | D²PO, Gemma2-Instruct (9B) · AlpacaEval 2 LC win rate | LC win rate (%): 59.7 -> 57.7 | Hurt |
| Replace exponential decay with power-law decay (ablation) | D²PO, Gemma2-Instruct (9B) · Arena-Hard win rate | win rate (%): 66.4 -> 71.2 | Helped |
| Set γ greater than 1.0, emphasizing later tokens (ablation) | D²PO · AlpacaEval 2 and Arena-Hard | win rate | Hurt |
Surprising findings
- Setting γ slightly above 1 (e.g., 1.01, 1.02) degrades performance below DPO, indicating overemphasis on later tokens is harmful.
- Off-policy reference-free SimPO collapses (near-zero win rates on Llama3-8B), while reference-free D²PO remains stable on on-policy data.
Limitations
- Gains are reported primarily with a single judge family (GPT-4 Turbo) and benchmark-specific win rates; off-policy reference-free variant is not used.
- Theoretical bound's tradeoff for choosing gamma is argued qualitatively; optimal gamma is selected empirically (0.98 best).
Experimental setup
Datasets: UltraFeedback
Benchmarks: AlpacaEval 2 Arena-Hard MT-Bench MMLU GSM8K MATH IFEval ARC-C HellaSwag TruthfulQA Winogrande
Metrics: win rate length-controlled win rate average MT-Bench score accuracy
Baselines: DPO KTO IPO SamPO ORPO SimPO SFT
Builds on: DPO SimPO SamPO IPO KTO
Closest papers by meaning
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.92)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.92)
- Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.92)
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.92)
- Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.92)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.