Research map / Large language models / Alignment and preferences
Self-Boosting Large Language Models with Synthetic Preference Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SynPO, an iterative self-boosting method that trains LLMs on fully synthetic preference pairs built from self-generated prompts and self-refined responses, without human preference annotation.
Problem
Collecting high-quality preference data for LLM alignment is costly and creativity-demanding, and static preference datasets limit continual improvement of LLMs.
Approach
A self-prompt generator (the LLM trained on keyword-to-prompt SFT) creates diverse prompts from keywords, and a response improver (the LLM trained to map its own outputs to gold responses) refines on-policy responses; pre- and post-refinement outputs form chosen/rejected pairs optimized with SimPO over iterations.</couldn't>
Key results
- After four SynPO iterations, Llama3-Base LC win rate on AlpacaEval 2.0 rises from 5.4% to 32.1% and Mistral-Base from 6.6% to 34.0%.
- SynPO improves Open LLM Leaderboard average by 3.19 points for Mistral-Base and 5.00 points for Llama3-Base over the SFT models.
- On MT-Bench, Llama3-Base SynPO Iter4 reaches 7.43 first-turn score versus 6.55 for SFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SynPO for four iterations instead of using the SFT model | Mistral-Base 7B · AlpacaEval 2.0 | length-controlled win rate (%): 6.6 -> 34.0 | Helped |
| Apply SynPO for three iterations instead of using the SFT model | Mistral-Base 7B · Arena-Hard | win rate (%): 2.0 -> 24.1 | Helped |
| Apply SynPO for four iterations instead of using the SFT model | Llama3-Base 8B · AlpacaEval 2.0 | length-controlled win rate (%): 5.4 -> 32.1 | Helped |
| Apply SynPO for three iterations instead of using the SFT model | Llama3-Base 8B · Arena-Hard | win rate (%): 2.7 -> 32.5 | Helped |
| Apply SynPO for four iterations instead of using the SFT model | Mistral-Base 7B · Open LLM Leaderboard | average score: 58.20 -> 61.39 | Helped |
| Apply SynPO for four iterations instead of using the SFT model | Llama3-Base 8B · Open LLM Leaderboard | average score: 63.34 -> 68.34 | Helped |
| Use SynPO-generated prompts instead of manually collected prompts (ablation) | SynPO on Llama3-8B · AlpacaEval 2.0 | length-controlled win rate (%): 23.8 -> 24.3 | Helped |
| Mix SynPO-generated prompts with manually collected prompts (ablation) | SynPO on Llama3-8B · AlpacaEval 2.0 | raw win rate (%): 24.5 -> 29.4 | Helped |
| Add preference optimization after seed-data fine-tuning (ablation) | Llama3-8B · AlpacaEval 2.0 | length-controlled win rate (%): 20.1 -> 24.6 | Helped |
| Optimize preferences directly on seed gold responses and initial model outputs (ablation) | Llama3-8B · AlpacaEval 2.0 | length-controlled win rate (%): 20.1 -> 11.6 | Hurt |
| Train seed-derived preference data for multiple epochs (ablation) | Llama3-8B · AlpacaEval 2.0 | length-controlled win rate (%): 24.6 -> 22.4 | Hurt |
| Apply SynPO across iterations | Mistral-Base · Open LLM Leaderboard GSM8k | accuracy: 34.72 -> 31.08 | Mixed |
Surprising findings
- Mistral-Base SynPO performance on GSM8K declines relative to SFT, while Llama3-Base improves by nearly 4 points.
- SynPO prompts alone outperform the UltraFeedback seed prompts they were derived from in the ablation.
Limitations
- Response improver trained on pseudo pairs from SFT data tends to learn paraphrasing rather than minimal edits.
- Relies on quality of a small seed high-quality dataset for validation and training the generators.
- Sensitive to scoring model choice; PairRM vs ArmoRM filtering yields different GSM8K outcomes.
Experimental setup
Datasets: UltraFeedback UltraChat-200k RefinedWeb
Benchmarks: AlpacaEval 2.0 Arena-Hard MT-Bench Open LLM Leaderboard LM Evaluation Harness tasks
Metrics: length-controlled win rate win rate MT-Bench score accuracy
Baselines: SFT UltraFeedback manual collection Sampling-Ranking Self-Rewarding Seed SFT Seed PO Seed SFT + PO
Builds on: SimPO Zephyr UltraFeedback Self-Rewarding LMs SPPO
Closest papers by meaning
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.90)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.90)
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.90)
- InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models (NeurIPS 2025 · similarity 0.90)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.90)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.