Research map / Large language models / Alignment and preferences
The Crucial Role of Samplers in Online Direct Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes gradient-descent convergence of DPO under different sampling distributions in tabular softmax bandits, showing online mixed samplers (DPO-R, DPO-P) reach quadratic convergence versus linear for uniform sampling, and proposes a practical logit-mixing posterior sampler evaluated on Safe-RLHF and Iterative-Prompt.
Problem
Convergence rates of DPO under different sampling strategies remain under-explored; prior theory on samplers relies on sample complexity and MLE oracles rather than optimization.
Approach
Treat DPO as gradient descent on a sampler-weighted loss; show that a reward-guided or policy-difference-guided mixed sampler (with inverse-sigmoid-derivative coefficients) cancels the linear term and yields quadratic convergence, whereas uniform sampling yields only linear rate. Practically, approximate the online sampler via posterior pi_theta^{2beta} and logit mixing of pi_ref and pi_theta.
Key results
- On Safe-RLHF at iteration 3, the proposed method reaches 80.7% test win-rate versus 73.3% for vanilla DPO.
- On Iterative-Prompt at iteration 3, the method reaches 84.4% test win-rate versus 79.0% for vanilla DPO.
- Theory: uniform sampler gives linear convergence (rate 0.588^T) while reward-guided and policy-guided mixed samplers give quadratic convergence (0.5^(2^T-1) and 0.611^(2^T-1)).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the uniform sampler with the reward-guided mixed sampler | Exact DPO · Tabular softmax bandits | Convergence rate: linear -> quadratic | Helped |
| Replace the uniform sampler with the policy-difference-guided mixed sampler | Exact DPO · Tabular softmax bandits | Convergence rate: linear -> quadratic | Helped |
| Use the practical posterior-based sampler with logit mixing instead of Vanilla DPO sampling | DPO · Safe-RLHF test, iteration 3 | Win-rate: 73.3 % -> 80.7 % | Helped |
| Use the practical posterior-based sampler with logit mixing instead of Vanilla DPO sampling | DPO · Iterative-Prompt test, iteration 3 | Win-rate: 79.0 % -> 84.4 % | Helped |
| Use the reward-guided mixed sampler with unbiased gradient estimates | Empirical DPO · Tabular softmax bandits | RMS preference-gap error: 14σ | Helped |
| Use the policy-difference-guided sampler with rejection sampling and unbiased gradient estimates | Empirical DPO · Tabular softmax bandits | RMS preference-gap error: 14σ | Helped |
Limitations
- Convergence analysis restricted to tabular softmax parametrization.
- No theoretical lower bound for the uniform sampler under noisy gradients.
- Evaluation relies on gold reward model scores rather than open benchmarks.
Experimental setup
Datasets: Safe-RLHF Iterative-Prompt
Benchmarks: Safe-RLHF Iterative-Prompt
Metrics: average gold reward win-rate against reference model reward-KL trade-off
Baselines: Vanilla DPO On-policy DPO Hybrid GSHF Online GSHF XPO
Builds on: DPO On-policy DPO Iterative DPO Hybrid GSHF Online GSHF
Closest papers by meaning
- Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization (NeurIPS 2025 · similarity 0.92)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.92)
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.91)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.91)
- TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights (ICLR 2025 · similarity 0.91)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.