Research map / Large language models / Alignment and preferences
Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sample Scheduling for DPO, where a contextual-bandit scheduler (SamS) picks a top-K subset of each training batch based on the policy's evolving hidden states and lagged DPO-loss rewards, improving DPO alignment without changing the DPO loss.
Problem
Fixed preference datasets contain noisy or misaligned samples whose difficulty shifts as the policy trains, and existing data selection methods are static pre-filters that ignore the evolving model state during DPO.
Approach
Formulate per-batch sample selection during DPO as a contextual bandit: an encoder over the policy's hidden states gives arm contexts, an exploitation MLP predicts a reward (batch-level loss reduction plus sample-level margin and uncertainty), and an exploration MLP adds a UCB-style bonus; the top-K samples per batch are used for the DPO update, with the reward collected in the next round at no ex
Key results
- On AlpacaEval 2 with Llama3-Instruct (8B), DPO+SamS reaches 42.2 LC / 40.5 WR versus 40.3 / 37.9 for DPO, and up to 51.5 LC on Llama3 v0.2.
- On Pythia-2.8B, DPO+SamS improves Anthropic-HH test accuracy from 64.3% to 67.1% and SHP from 67.6% to 70.0%.
- With 20% label noise on Anthropic-HH, DPO+SamS converges to about 64% versus about 58% for DPO; SamS reduces GPU memory by about 18% at similar runtime.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adaptive batch-wise sample scheduling to DPO | DPO · AlpacaEval 2 across the four reported model configurations | LC win rate: 5.5% - 8.4% | Helped |
| add adaptive batch-wise sample scheduling to DPO | DPO · AlpacaEval 2 across the four reported model configurations | win rate: 3.0% - 12.4% | Helped |
| add adaptive sample scheduling | DPO · HH test set | test accuracy: +2.8 | Helped |
| add adaptive sample scheduling | DPO · SHP test set | test accuracy: +2.4 | Helped |
| add adaptive sample scheduling | KTO · HH test set | test accuracy: +3.1 | Helped |
| add adaptive sample scheduling | KTO · SHP test set | test accuracy: +2.3 | Helped |
| select 75% rather than 100% of each batch (ablation) | SamS with DPO · HH test set | test accuracy: 64% -> 68% | Helped |
| select 50% rather than 100% of each batch (ablation) | SamS with DPO · HH test set | test accuracy: 64% -> 67% | Helped |
| select 25% rather than 100% of each batch (ablation) | SamS with DPO · HH test set | test accuracy: 64% -> 63% | Hurt |
| include the exploration network fS' (ablation) | SamS with DPO · HH test set | test accuracy: 65% -> 67% | Helped |
Surprising findings
- SamS gains shrink when higher-quality preference data is available (v0.2 setting), per the authors' limitations discussion.
- Selecting only 25% of samples per batch with SamS underperforms standard DPO on the Anthropic-HH ablation.
Limitations
- Performance depends on data quality; benefits diminish when high-quality preference pairs are abundant (v0.2 setting).
- Evaluated only in offline DPO setting; online iterative DPO not studied.
- Ablation and noise experiments are on a single small model and dataset setting.
Experimental setup
Datasets: UltraFeedback Anthropic-HH SHP AlpacaEval 2 MT-Bench
Benchmarks: AlpacaEval 2 MT-Bench Anthropic-HH test accuracy SHP test accuracy
Metrics: length-controlled win rate raw win rate MT-Bench score (GPT-4 Turbo judge) test accuracy GPU memory runtime
Baselines: RRHF SLiC-HF IPO CPO KTO ORPO R-DPO DPO (50% random) Selective DPO
Builds on: DPO Selective DPO Online Iterative DPO UCB Thompson Sampling
Closest papers by meaning
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.93)
- The Crucial Role of Samplers in Online Direct Preference Optimization (ICLR 2025 · similarity 0.92)
- Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.92)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.91)
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.91)
- Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.