Research map / Large language models / Alignment and preferences
Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NS-DPO, a single-discount-parameter extension of DPO that exponentially down-weights older preference data under a Dynamic Bradley-Terry model, with theoretical regret bounds and experiments under synthetic and LLM preference drift.
Problem
Standard preference optimization assumes stationary human preferences; when preferences drift over time in offline datasets, stationary methods treat outdated labels as noise and learn misaligned policies.
Approach
Model time-varying rewards with a Dynamic Bradley-Terry model and fit the DPO objective with exponential time-discount weights gamma^(T-t_i-1), so recent datapoints dominate learning; the only added hyperparameter is gamma, with regret bounds under a variation-budget assumption.
Key results
- On UltraFeedback-LM with Llama-3.2-1B, NS-DPO achieves LCWR of 8.93 vs DPO 7.29 at rho=0.7, t_cp=21, and 7.36 vs 6.49 at rho=1.0, t_cp=81 (AlpacaEval2).
- NS-DPO outperforms DPO in reward accuracy by over 10% on TV-HH with gradual drift and on 2C NSGO shows a gap of nearly 10% in reward accuracy.
- Under stationary UltraFeedback (t_cp=0), NS-DPO matches DPO reward accuracy and achieves higher LCWR (9.12 vs 8.81).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use exponentially discounted preference data (NS-DPO) instead of stationary DPO | Llama-3.2-1b-it · UltraFeedback-LM, sudden preference drift | LCWR: 7.29 -> 8.93; 7.85 -> 8.38; 7.17 -> 7.85; 8.23 -> 9.00; 6.99 -> 7.41; 6.49 -> 7.36 | Helped |
| Use exponentially discounted preference data (NS-DPO) instead of sliding-window DPO | Llama-3.2-1b-it · UltraFeedback-LM, sudden preference drift | LCWR: 6.09 -> 8.93; 4.93 -> 8.38; 4.63 -> 7.85; 6.71 -> 9.00; 5.59 -> 7.41; 4.83 -> 7.36 | Helped |
| Use NS-DPO rather than DPO when there is no preference drift | Llama-3.2-1b-it · UltraFeedback-LM (no drift) | LCWR: 8.81 -> 9.12 | Helped |
| Discount older preference examples with NS-DPO rather than using stationary preference optimization | Llama-2-7b-chat-hf · 2C NSGO, gradual opinion drift | reward accuracy: nearly 10% | Helped |
| Append time-step information to prompts (tDPO) instead of using stationary DPO | Llama-2-7b-chat-hf · 2C NSGO | reward accuracy | No clear effect |
| Use NS-DPO rather than DPO for gradual preference drift | Llama-2-7b-chat-hf · TV-HH | reward accuracy: over $ 10\% | Helped |
| Use NS-DPO rather than DPO when the preference change point is early | Llama-3.2-1b-it · TV-HH (tcp ∈ {10, 30}) | win rate | No clear effect |
| Use NS-DPO rather than DPO when the preference change point is late | Llama-3.2-1b-it · TV-HH | win rate: >0.5 | Helped |
| Use NS-DPO rather than stationary DPO | Llama-2-7b-chat-hf · UltraFeedback (no preference drift) | reward accuracy | No clear effect |
| Increase the discount factor γ (ablation) | NS-DPO · Synthetic preference-drift experiment | training steps to final test accuracy | Helped |
| Set γ very close to 1 (γ > 0.97) (ablation) | NS-DPO · Synthetic preference-drift experiment | reward accuracy | Hurt |
| Vary γ across 0.5 to 0.9 (ablation) | NS-DPO · Synthetic preference-drift experiment | final test accuracy: 0.5 to 0.9 | No clear effect |
Surprising findings
- NS-DPO matches or slightly exceeds DPO even when no preference drift is present, so the discount does not hurt stationary performance.
- In-context time-step conditioning (tDPO) gives no meaningful improvement over stationary DPO.
Limitations
- Assumes a known upper bound B_T on total preference drift; gamma depends on it.
- Theory limited to offline setting with log-linear policies; LLM experiments use the synthetic/constructed non-stationary datasets rather than real temporal preference data.
- Discounting also downweights stationary datapoints, and performance depends on the choice of gamma.
Experimental setup
Datasets: UltraFeedback HH-RLHF harmless-base GlobalOpinionsQA
Benchmarks: AlpacaEval2 TV-HH test split UltraFeedback-RM test split 2C NSGO
Metrics: reward accuracy win rate length-controlled win rate
Baselines: DPO IPO tDPO SW-DPO
Builds on: Direct Preference Optimization Identity Preference Optimization Bradley-Terry model discounted UCB / exponentially weighted estimation for non-stationary bandits
Closest papers by meaning
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.90)
- KL Penalty Control via Perturbation for Direct Preference Optimization (NeurIPS 2025 · similarity 0.90)
- Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.89)
- Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.89)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.