Research map / Large language models / Alignment and preferences
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Magpie synthesizes large-scale instruction-response alignment data by feeding only the pre-query chat template to aligned LLMs (e.g., Llama-3-Instruct), letting them self-generate user queries and then responses, without seeds or prompt engineering.
Problem
Open alignment datasets are scarce because human annotation is costly and prompt-based synthetic generation from seed questions lacks scalability and diversity; the paper asks whether high-quality instruction data can be extracted directly from aligned LLMs.
Approach
Input only the pre-query template of an aligned LLM so its auto-regressive generation produces a user instruction; then feed that instruction back with the assistant template to generate the response. Scales to millions of pairs with no seed questions, and extends to multi-turn, DPO (RM-ranked samples), domain-specific, and multilingual data via system prompts.
Key results
- Magpie-Air-300K-Filtered SFT on Llama-3-8B reaches 49.27% LC win rate on AlpacaEval 2 (vs GPT-4-Turbo), exceeding all SFT baselines (best baseline WildChat 34.85%) and the official Llama-3-8B-Instruct (SFT+DPO) at 22.92%.
- Magpie-Pro-DPO on Llama-3-8B reaches 50.10% LC win rate vs GPT-4-Turbo on AlpacaEval 2, outperforming GPT-4-Turbo (1106) reference setting in the paper's claim.
- Magpie-Pro-300K-Filtered achieves 25.08% LC / 29.47% WR vs GPT-4-Turbo and 52.12% LC vs Llama-3-8B-Instruct, with 300K SFT data versus >10M for the official model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| SFT on Magpie data (self-synthesized, no seed questions) instead of public baselines | Llama-3-8B base · AlpacaEval 2 and Arena-Hard | LC and WR | Helped |
| Filtering the Magpie-Pro 300K data (quality, difficulty, reward filters, longest responses) (ablation) | Magpie-Pro SFT · AlpacaEval 2 vs GPT-4-Turbo | LC (%): 21.65 -> 25.08 LC | Helped |
| Filtering the Magpie-Air 300K data (ablation) | Magpie-Air SFT · AlpacaEval 2 vs GPT-4-Turbo | LC (%): 21.99 -> 22.66 LC | Helped |
| Adding Magpie DPO data after SFT (SFT+DPO) | Magpie-Pro SFT model · AlpacaEval 2 vs GPT-4-Turbo | LC (%): 25.08 -> 50.10 LC | Helped |
| Magpie-Pro-DPO model (SFT+DPO) versus official Llama-3-8B-Instruct | Llama-3-8B · Arena-Hard | WR (%): 35.7 vs 20.6 | Helped |
| Adding 150K math/code/reasoning booster data to Pro-300K-Filtered (ablation) | Magpie-Pro SFT · GSM8K (5-shot) | accuracy: 47.92 -> 63.08 | Helped |
| Adding reasoning booster data to Pro-300K-Filtered (ablation) | Magpie-Pro SFT · Open LLM average (7 benchmarks) | Average: 61.58 -> 64.21 | Helped |
| Using Magpie-Pro-300K-Filtered SFT on Qwen2-1.5B base vs official instruct model | Qwen2-1.5B · AlpacaEval 2 vs GPT-4-Turbo | LC (%): 3.48 vs 3.91 (LC); WR 5.32 vs 3.00 | Helped |
| Magpie-Pro-300K-Filtered SFT on Qwen1.5-7B base | Qwen1.5-7B · AlpacaEval 2 vs GPT-4-Turbo | LC (%): 15.10 vs 14.75 (LC); WR 18.51 vs 11.77 | No clear effect |
| Magpie-generated data vs Self-Instruct using same Llama-3-8B-Instruct generator (100K) (ablation) | Llama-3-8B base SFT · AlpacaEval 2 vs Llama-3-8B-Instruct | LC (%): 17.86 -> 46.82 LC | Helped |
| Multi-turn extension (MT) vs single-turn Magpie-Air (ablation) | Magpie-Air SFT · Arena-Hard | WR (%): 14.9 -> 15.5 | Helped |
| Multi-turn extension (MT) vs single-turn Magpie-Pro (ablation) | Magpie-Pro SFT · Arena-Hard | WR (%): 18.9 -> 20.4 | Helped |
Surprising findings
- Instruction-only prompting with the pre-query template yields high-quality, diverse queries without seeds, and the paper hypothesizes LLMs implicitly memorize instruction distributions even when instruction loss is masked.
- Adding the default Vicuna system prompt during instruction generation lowered quality and made instructions easier.
Limitations
- Performance degrades on math and reasoning benchmarks due to few reasoning instructions; a booster dataset partially addresses this.
- Up to about 1% of data flagged as potentially harmful by Llama-Guard-2, mostly specialized advice.
- Relies on LLM judges (GPT-4-Turbo) for AlpacaEval and Arena-Hard, and the reason for the memorization effect is untested.
Experimental setup
Datasets: ShareGPT WildChat Evol-Instruct UltraChat OpenHermes 1 OpenHermes 2.5 Tulu V2 Mix GenQA UltraFeedback
Benchmarks: AlpacaEval 2 Arena-Hard WildBench MMLU ARC HellaSwag TruthfulQA WinoGrande GSM8K MMLU-Redux
Metrics: length-controlled win rate win rate accuracy
Baselines: ShareGPT WildChat Evol-Instruct UltraChat OpenHermes 2.5 Tulu V2 Mix GenQA UltraFeedback (DPO) Self-Instruct (Llama-3)
Builds on: DPO Self-Instruct URIAL
Closest papers by meaning
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.90)
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.90)
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.89)
- Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples (ICML 2025 · similarity 0.89)
- MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models (NeurIPS 2026 · similarity 0.89)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.