Research map / Large language models / Alignment and preferences
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows on-policy and off-policy preference data are complementary across task types and proposes SimpleMix, which trains DPO on a 50/50 mix of on-policy (SFT-sampled, reward-labeled) and off-policy (UltraFeedback/HelpSteer2) pairs, beating on-policy, off-policy, HyPO and DPO-Mix-P.
Problem
Prior work disagrees on whether on-policy preference data beats off-policy data for LM preference optimization, and existing comparisons do not control the alignment algorithm or break results down by task type.
Approach
Hold DPO fixed and vary only the data source; find on-policy data helps verifiable tasks (math, coding) and off-policy helps open-ended tasks, then simply mix 50/50 on-policy and off-policy pairs, which beats more complex hybrid methods.
Key results
- SimpleMix improves over on-policy DPO and off-policy DPO by an average of 6.03% length-controlled win rate on Alpaca Eval 2.0 (per abstract).
- On Llama-3.1-8B-Instruct with UltraFeedback, SimpleMix reaches 29.41 LC win rate vs 27.28 for on-policy DPO and 27.87 for off-policy DPO; Avg. Score 63.06 vs 62.09.
- SimpleMix outperforms HyPO and DPO-Mix-P by an average of 3.05% on Alpaca Eval 2.0 (per abstract).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Mix on-policy and off-policy preference data with equal probability | DPO · Alpaca Eval 2.0 | Length-controlled win rate: 6.03% | Helped |
| Mix on-policy and off-policy preference data | DPO · Alpaca Eval 2.0 | Length-controlled win rate: 3.05% | Helped |
| Use on-policy instead of off-policy preference data | DPO · Alpaca Eval 2.0 Math | Length-controlled win rate: +5.72% | Helped |
| Use on-policy instead of off-policy preference data | DPO · Alpaca Eval 2.0 Coding | Length-controlled win rate: +7.02% | Helped |
| Use on-policy instead of off-policy preference data | DPO · Alpaca Eval 2.0 Creative writing | Length-controlled win rate: -2.85% | Hurt |
| Use on-policy instead of off-policy preference data | DPO · Alpaca Eval 2.0 Recommendation | Length-controlled win rate: -3.29% | Hurt |
| Elicit diverse responses by prompting iteratively rather than mixing data sources (ablation) | DPO on Llama-3.1-8B-Instruct · Alpaca Eval 2.0 | LC (%): 29.41 -> 21.41 | Hurt |
| Increase sampling temperature to τ = 1.0 rather than use SIMPLEMIX (ablation) | DPO on Llama-3.1-8B-Instruct · Alpaca Eval 2.0 | LC (%): 29.41 -> 26.94 | Hurt |
| Increase sampling temperature to τ = 2.0 rather than use SIMPLEMIX (ablation) | DPO on Llama-3.1-8B-Instruct · Alpaca Eval 2.0 | LC (%): 29.41 -> 26.77 | Hurt |
| Increase sampling temperature to τ = 3.0 rather than use SIMPLEMIX (ablation) | DPO on Llama-3.1-8B-Instruct · Alpaca Eval 2.0 | LC (%): 29.41 -> 26.18 | Hurt |
| Use a balanced 0.5 on-policy + 0.5 off-policy mixture instead of other mixture ratios (ablation) | DPO · UltraFeedback | Alpaca Eval 2.0 win rate | Helped |
| Filter off-policy data by reward-model quality before mixing with on-policy data (ablation) | SIMPLEMIX · Alpaca Eval 2.0 | Length-controlled win rate | Helped |
Surprising findings
- On-policy and off-policy DPO perform nearly identically on aggregate benchmarks, with the gap minimal, contrary to prior claims that on-policy data consistently wins.
- Increasing response diversity via prompting or higher sampling temperature lowers reward quality and hurts downstream performance.
Limitations
- Limited hyperparameter search (only beta=0.1 and learning rate 5e-7 reported for all runs), so conclusions may not hold under other configurations.
- LLM-as-a-judge evaluation (Alpaca Eval 2.0) may be hackable and biased.
- Only two 8B models and two preference datasets tested.
Experimental setup
Datasets: UltraFeedback HelpSteer2 Alpaca Eval 2.0 FineWeb evaluation suite
Benchmarks: Alpaca Eval 2.0 ARC-Challenge CommonsenseQA HellaSwag MMLU OpenBookQA PIQA Winogrande IFEval
Metrics: length-controlled win rate accuracy prompt-level loose accuracy average score
Baselines: On-policy DPO Off-policy DPO HyPO DPO-Mix-P
Builds on: DPO HyPO DPO-Mix-P PPO
Closest papers by meaning
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.90)
- Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.90)
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.90)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.90)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.90)
- MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.