Research map / Image generation / Text-to-image diffusion
Self-Play Fine-tuning of Diffusion Models for Text-to-image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPIN-Diffusion, a self-play fine-tuning method for text-to-image diffusion models that needs only one image per prompt, and shows gains over SFT and Diffusion-DPO on Stable Diffusion 1.5 using Pick-a-Pic.
Problem
Fine-tuning text-to-image diffusion models usually needs paired winner/loser preference data (for RL or DPO) or plateaus under SFT; many real datasets have only one image per prompt.
Approach
Adapts LLM self-play fine-tuning (SPIN) to diffusion models: the main model is trained to separate real (winner) diffusion trajectories from trajectories sampled from the previous iteration's copy, using a decomposed DDIM-based objective that uses only consecutive timesteps, so no loser images are needed.
Key results
- SPIN-Diffusion Iter3 reaches average score 7.4126 on Pick-a-Pic test vs 7.1726 for Diffusion-DPO and 7.0133 for SD-1.5 (mean scores).
- At iteration 2, SPIN-Diffusion wins 90.0% against SD-1.5 on average vs 73.2% for SFT and 84.8% for Diffusion-DPO (ours).
- Iteration 1 already beats SFT on Aesthetic (6.12 vs 5.95) and PickScore (21.67 vs 21.45).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPIN-Diffusion instead of SFT | Stable Diffusion 1.5 · Pick-a-Pic test set | PickScore: 21.4542 -> 21.6651 | Helped |
| Use SPIN-Diffusion iteration 2 instead of Diffusion-DPO (ours) | Stable Diffusion 1.5 · Pick-a-Pic test set | Average: 7.2758 -> 7.3951 | Helped |
| Continue SPIN-Diffusion from iteration 2 to iteration 3 | SPIN-Diffusion · Pick-a-Pic test set | Average: 7.3951 -> 7.4126 | Helped |
| Use SPIN-Diffusion iteration 2 instead of Diffusion-DPO | Stable Diffusion 1.5 · PartiPrompts | Average: 7.3698 -> 7.4326 | Helped |
| Use SPIN-Diffusion iteration 2 instead of Diffusion-DPO | Stable Diffusion 1.5 · HPSv2 | Average: 7.4388 -> 7.5095 | Helped |
| Continue training iteration 1 without updating the opponent; instead update the opponent to iteration 1 for iteration 2 (ablation) | SPIN-Diffusion · Pick-a-Pic validation set | PickScore | Helped |
| Continue training iteration 1 without updating the opponent (ablation) | SPIN-Diffusion · Pick-a-Pic validation set | HPS | Hurt |
| Continue training iteration 1 without updating the opponent (ablation) | SPIN-Diffusion · Pick-a-Pic validation set | Average score | Hurt |
| Continue training iteration 1 without updating the opponent (ablation) | SPIN-Diffusion · Pick-a-Pic validation set | Aesthetic | Mixed |
| Use SPIN-Diffusion iteration 3 instead of SD-1.5 | Stable Diffusion 1.5 · Pick-a-Pic test set | Overall winning rate: 91.6% vs 73.2% | Helped |
Surprising findings
- SFT plateaus after about 50k training examples, while SPIN-Diffusion keeps improving, and continued training within iteration 1 degrades PickScore and HPS, suggesting opponent updates matter more than extra epochs.
Limitations
- Evaluated only on SD-1.5 scale models with Pick-a-Pic data, which is generated by Dreamlike and SDXL-beta rather than human-collected single-image data
- Relies on automatic reward models (PickScore, HPS, ImageReward, Aesthetic) and no human evaluation is reported
- Approximate objective uses an upper bound (Jensen) and approximations of the backward process
Experimental setup
Datasets: Pick-a-Pic LAION-5B
Benchmarks: Pick-a-Pic test set PartiPrompts HPSv2
Metrics: PickScore HPS ImageReward Aesthetic score winning rate
Baselines: SD-1.5 SFT Diffusion-DPO
Builds on: SPIN DDIM Diffusion-DPO Stable Diffusion
Closest papers by meaning
- Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.90)
- Directly Fine-Tuning Diffusion Models on Differentiable Rewards (ICLR 2024 · similarity 0.90)
- Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · similarity 0.90)
- David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.90)
- Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model (CVPR 2024 · similarity 0.89)
- Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.