Research map / Image generation / Text-to-image diffusion
SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SwiftBrush, an image-free one-step text-to-image distillation method that adapts variational score distillation from text-to-3D to distill Stable Diffusion 2.1 into a single-step student using only text prompts.
Problem
Text-to-image diffusion models are slow due to iterative sampling, and prior one-step distillation methods depend heavily on large numbers of real or teacher-generated images and are complex to train.
Approach
Replace the NeRF renderer in text-to-3D variational score distillation with a one-step text-to-image student network, train it with only text prompts (no images) using a frozen Stable Diffusion teacher plus a trainable LoRA teacher, and re-parameterize the student output to predict x0.
Key results
- SwiftBrush achieves FID 16.67 and CLIP score 0.29 on COCO-30K in one step, versus InstaFlow's 13.10 FID and LCM's 35.56 FID.
- On HPSv2, SwiftBrush scores highest among one-step methods across anime, photo, concept art and paintings (e.g., 27.21 photo).
- A user study with 400 users over 30 prompts found SwiftBrush preferred over InstaFlow on fidelity and accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SwiftBrush instead of InstaFlow | One-step text-to-image generation · COCO 2014 30K | CLIP-30K: 0.28§ -> 0.29 | Helped |
| Use SwiftBrush instead of InstaFlow | One-step text-to-image generation · COCO 2014 30K | FID-30K: 13.10† -> 16.67 | Hurt |
| Use SwiftBrush instead of BOOT | One-step text-to-image generation · COCO 2014 30K | FID-30K: 48.20 -> 16.67 | Helped |
| Use SwiftBrush instead of BOOT | One-step text-to-image generation · COCO 2014 30K | CLIP-30K: 0.26 -> 0.29 | Helped |
| Use SwiftBrush instead of LCM | One-step text-to-image generation · HPSv2 Anime | Human Preference Score v2: 22.61 -> 26.91 | Helped |
| Use SwiftBrush instead of LCM | One-step text-to-image generation · HPSv2 Photo | Human Preference Score v2: 22.71 -> 27.21 | Helped |
| Use SwiftBrush instead of LCM | One-step text-to-image generation · HPSv2 Concept Art | Human Preference Score v2: 22.74 -> 26.32 | Helped |
| Use SwiftBrush instead of LCM | One-step text-to-image generation · HPSv2 Paintings | Human Preference Score v2: 22.91 -> 26.37 | Helped |
| Add the LoRA teacher during student training (ablation) | SwiftBrush · COCO 2014 5K subset | FID | Helped |
| Add the LoRA teacher during student training (ablation) | SwiftBrush · COCO 2014 5K subset | CLIP score | Helped |
| Use LoRA rank 64 instead of rank 4 (ablation) | SwiftBrush LoRA teacher · COCO 2014 5K subset | FID | Helped |
| Re-parameterize the student output as predicted x0 (ablation) | SwiftBrush · COCO 2014 5K subset | FID | Helped |
Surprising findings
- Without the LoRA teacher (plain SDS), the student produces overly saturated, unrealistic images, and a 4-rank LoRA causes mode collapse after about 8K steps.
Limitations
- Single-step student has lower quality than the multi-step teacher.
- Cannot currently support few-step generation.
- FID is worse than InstaFlow (16.67 vs 13.10) and SD 25-step (13.45).
Experimental setup
Datasets: JourneyDB captions MS COCO 2014 validation
Benchmarks: COCO 2014 30K HPSv2
Metrics: FID CLIP score HPSv2 score
Baselines: Guided Distillation LCM InstaFlow BOOT Stable Diffusion 2.1 (25 steps)
Builds on: ProlificDreamer Score Distillation Sampling Latent Diffusion Models LoRA
Closest papers by meaning
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.90)
- Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR 2024 · similarity 0.90)
- Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generation (ICLR 2025 · similarity 0.90)
- Your Student is Better Than Expected: Adaptive Teacher-Student Collaboration for Text-Conditional Diffusion Models (CVPR 2024 · similarity 0.90)
- InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (ICLR 2024 · similarity 0.90)
- SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.