Research map / Large language models / Instruction and fine-tuning
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPIN, a self-play fine-tuning method that turns an SFT model into a stronger one by training it to distinguish its own previous-iteration generations from human SFT data, without extra annotations.
Problem
Improving an LLM beyond its SFT checkpoint without acquiring additional human-annotated or AI-feedback preference data; further SFT on the same data plateaus or degrades performance.
Approach
A self-play game in which the new LLM iterate is trained with a logistic-loss objective to prefer human SFT responses over responses sampled from the previous iterate (the opponent), with the opponent updated to the new model each round; the objective has a closed-form optimum at the data distribution, and resembles DPO but uses only SFT data.
Key results
- SPIN raises zephyr-7b-sft-full average on Open LLM Leaderboard from 58.14 to 63.16 by iteration 3, with GSM8k and TruthfulQA gains above 10% and 5%.
- SPIN iteration 0 reaches average 60.80, comparable to zephyr-7b-beta (DPO with 62k GPT-4-labeled preferences, average 61.31).
- On MT-Bench, SPIN improves from 5.94 to 6.78 at iteration 2, surpassing vicuna-13b-v1.5 (6.57).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SPIN self-play fine-tuning through iteration 3 | zephyr-7b-sft-full · HuggingFace Open LLM Leaderboard | Average: 58.14 -> 63.16 | Helped |
| Apply SPIN self-play fine-tuning through iteration 3 | zephyr-7b-sft-full · GSM8K | acc: 26.76 -> 38.97 | Helped |
| Apply SPIN self-play fine-tuning through iteration 3 | zephyr-7b-sft-full · TruthfulQA | mc2: 43.73 -> 54.90 | Helped |
| Apply SPIN self-play fine-tuning through iteration 3 | zephyr-7b-sft-full · Winogrande | acc: 74.19 -> 73.72 | Hurt |
| Apply SPIN self-play fine-tuning through iteration 3 | zephyr-7b-sft-full · BB-sports | accuracy: 96.0 -> 94.4 | Hurt |
| Apply SPIN self-play fine-tuning through iteration 2 | zephyr-7b-sft-full · MT-Bench | score: 5.94 -> 6.78 | Helped |
| Continue SFT for one more epoch on Ultrachat200k (ablation) | zephyr-7b-sft-full · HuggingFace Open LLM Leaderboard | Average: 58.14 -> 57.23 | Hurt |
| Use iterative SPIN training instead of extending training within iteration 0 (ablation) | zephyr-7b-sft-full · HuggingFace Open LLM Leaderboard | Average | Helped |
| Increase SPIN training size from 14k to 50k (ablation) | SPIN iteration 0 · HuggingFace Open LLM Leaderboard | Average | Helped |
| Fine-tune SPIN iteration 3 with DPO for two epochs on 62k preference examples | SPIN iteration 3 · HuggingFace Open LLM Leaderboard | Average: 63.16 -> 64.05 | Helped |
| Fine-tune SPIN iteration 3 with DPO for two epochs on 62k preference examples | SPIN iteration 3 · TruthfulQA | mc2: 54.90 -> 60.07 | Helped |
| Use SPIN iteration 1 rather than DPO training | zephyr-7b-sft-full · HuggingFace Open LLM Leaderboard | Average: 61.31 -> 62.12 | Helped |
Surprising findings
- Further SFT for one more epoch on the same Ultrachat200k data degraded the average score from 58.14 to 57.23.
- Training for more epochs within iteration 0 plateaus, so gains require new iterations rather than longer training.
Limitations
- Performance is capped by the fixed human target data distribution, so gains cannot exceed the SFT data quality.
- Requires generating synthetic data each iteration, which is resource-intensive.
Experimental setup
Datasets: UltraChat200k UltraFeedback Binarized
Benchmarks: HuggingFace Open LLM Leaderboard (ARC, TruthfulQA, Winogrande, GSM8k, HellaSwag, MMLU) MT-Bench BIG-Bench Hard (causal judgment, formal fallacies, sports understanding) OpenBookQA
Metrics: accuracy acc_norm mc2 average score MT-Bench score
Baselines: SFT (zephyr-7b-sft-full) DPO (zephyr-7b-beta) vicuna-13b-v1.5
Builds on: DPO zephyr-7b-sft-full Alignment Handbook AlphaGo Zero Wasserstein GAN
Closest papers by meaning
- Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs (NeurIPS 2025 · similarity 0.94)
- IRIS: Interpolative Rényi Iterative Self-play for Large Language Model Fine-Tuning (NeurIPS 2026 · similarity 0.93)
- SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models (NeurIPS 2025 · similarity 0.90)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.90)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.89)
- From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.