Research map / Image generation / Fast diffusion sampling
Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a standalone GAN objective (with a Projected GAN discriminator) turns pre-trained EDM diffusion models into one-step generators, and that freezing most convolutional layers (D2O-F) keeps performance, supporting a view of diffusion training as generative pre-training.
Problem
Diffusion distillation into one-step generators is expensive and often underperforms the teacher, and the mechanism by which GAN losses help distillation is unclear.
Approach
Argues that teacher and student converge to different local minima under instance-level distillation, so a pure GAN objective (no distillation loss) fine-tuning a pre-trained EDM U-Net works better; freezing most convolutional weights (D2O-F) shows diffusion pre-training already provides one-step generative capacity.
Key results
- D2O-F reaches FID 1.54 on CIFAR-10 unconditional with 5M training images, and FID 1.16 on ImageNet 64x64 with 5M images.
- D2O-F with only 0.2M training images achieves FID 4.12 on CIFAR-10 (per text), while D2O reaches FID 3.62 with 0.2M images in the ablation section.
- Freezing 85.8% of parameters (convolutional layers) gives FID 1.54 vs 1.66 when all layers are tuned on CIFAR-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the PG discriminator with VGG16 (ablation) | D2O · CIFAR-10 | FID: 2.21 -> 4.04 | Hurt |
| Replace the PG discriminator with LPIPS (ablation) | D2O · CIFAR-10 | FID: 2.21 -> 3.49 | Hurt |
| Enable augmentation (ablation) | D2O with PG discriminator · CIFAR-10 | FID: 1.98 -> 2.21 | Hurt |
| Add R1 regularization with γ=1e-4 and disable augmentation (ablation) | D2O with PG discriminator · CIFAR-10 | FID: 1.98 -> 1.66 | Helped |
| Freeze convolution layers while tuning normalization, QKV, and skip layers (ablation) | D2O-F · CIFAR-10 | FID: 1.66 -> 1.54 | Helped |
| Also freeze QKV projections (ablation) | D2O-F with frozen convolution layers · CIFAR-10 | FID: 1.54 -> 1.60 | Hurt |
| Also freeze skip layers (ablation) | D2O-F with frozen convolution and QKV layers · CIFAR-10 | FID: 1.60 -> 2.51 | Hurt |
| Add the Consistency Distillation loss to the GAN objective (ablation) | D2O and D2O-F · CIFAR-10 | convergence | Hurt |
| Freeze convolutional layers during distillation (ablation) | Consistency Distillation · CIFAR-10 | FID | Hurt |
| Use D2O-F instead of D2O | D2O · AFHQv2 64x64 | FID: 1.23 -> 1.31 | Hurt |
| Use D2O-F instead of D2O | D2O · FFHQ 64x64 | FID: 1.08 -> 0.85 | Helped |
| Use D2O-F instead of D2O | D2O · ImageNet 64x64 class-conditional | FID: 1.42 -> 1.16 | Helped |
Surprising findings
- Freezing most convolutional layers improves rather than degrades one-step GAN fine-tuning, while freezing layers hurts CD-based distillation (CD-F performs poorly).
- Adding an extra consistency distillation loss slows convergence and gives no benefit over D2O-F.
Limitations
- Only evaluated on U-Net-based EDM architecture; DiT not tested
- Only tested on 64x64 resolution datasets; higher resolution and text-to-image not explored
- Frequency analysis described as preliminary
Experimental setup
Datasets: CIFAR-10 AFHQv2 64x64 FFHQ 64x64 ImageNet 64x64
Benchmarks: CIFAR-10 AFHQv2 64x64 FFHQ 64x64 ImageNet 64x64
Metrics: FID Inception Score precision recall CLIP-FID
Baselines: Consistency Distillation CTM DMD SiD iCT BOOT DMD2 ECM
Builds on: EDM Projected GAN Consistency Distillation StyleGAN2-ADA
Closest papers by meaning
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.91)
- One-Step Diffusion Distillation through Score Implicit Matching (NeurIPS 2024 · similarity 0.91)
- You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs (ICLR 2025 · similarity 0.90)
- Neural Diffusion Models (ICML 2024 · similarity 0.90)
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.90)
- Simple and Fast Distillation of Diffusion Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.