Research map / Image generation / Fast diffusion sampling
Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SiDA, which adds a discriminator loss reusing the fake score network's encoder to Score identity Distillation (SiD), enabling one-step data-aware distillation that surpasses EDM/EDM2 teachers on CIFAR-10, ImageNet 64x64 and ImageNet 512x512.
Problem
Data-free one-step diffusion distillation (SiD) is bottlenecked by the teacher's inaccuracy in representing true data scores, so the distilled generator cannot exceed the teacher's quality.
Approach
Reuse the encoder of the fake score network as a patch-level discriminator on real vs. generated diffused images, adding a per-GPU-batch-normalized adversarial loss to the SiD loss with no extra parameters; SiD^2A further initializes from a pre-distilled SiD generator.
Key results
- SiD^2A achieves FID 1.499 on unconditional CIFAR-10 and 1.110 on ImageNet 64x64 in one step, beating the teachers and prior distillation methods.
- On ImageNet 512x512, SiD^2A reaches FID 1.366 for EDM2-XXL in one step without CFG, versus 1.81 for the EDM2-XXL teacher with CFG and 63 steps.
- SiDA converges roughly an order of magnitude faster than SiD, needing ~10M synthesized images versus ~100M for SiD to match the teacher on EDM2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adversarial loss to SiD | SiD distillation · CIFAR-10 unconditional | FID: 2.028 $\pm$ -> {{1.516}} $\pm$ | Helped |
| Add adversarial loss to SiD | SiD distillation · ImageNet 512x512 across EDM2 sizes | FID: SiD: 3.353, 2.707, 2.060, 1.907, 1.888, {1.969}; SiDA: 2.228, 1.756, 1.546, 1.501, {1.428}, {1.503} | Helped |
| Initialize SiDA from a distilled SiD generator | SiDA (SiD²A) · CIFAR-10 unconditional | FID: 1.516 -> 1.499 | Helped |
| Initialize SiDA from a distilled SiD generator | SiDA (SiD²A) · CIFAR-10 conditional | FID: 1.436 -> 1.396 | Helped |
| Initialize SiDA from a distilled SiD generator | SiDA (SiD²A) · ImageNet 64x64 | FID: 1.353 -> 1.110 | Helped |
| Use α=1.0 rather than lower values or α values 1.2 and 1.5 (ablation) | SiDA · CIFAR-10 unconditional | FID | Mixed |
| Initialize the generator from a pre-distilled SiD generator (ablation) | SiD²A · EDM distillation across tested datasets | training stability | Helped |
| Remove the SiD component, leaving only adversarial loss (ablation) | SiDA generator loss · Single-step distillation | training stability | Hurt |
| Replace in-place forced weight normalization with pre-forward-hook normalization (ablation) | SiDA on EDM2 · EDM2-XS, EDM2-S, and EDM2-M | FID | No clear effect |
| Disable the data-specific one-layer MLP for loss weights (ablation) | SiDA on EDM2 · ImageNet 512x512 | FID | No clear effect |
| Add adversarial loss to SiD | SiD distillation · EDM2 distillation convergence | images needed to exceed teacher performance: approximately 100M -> 10M | Helped |
Surprising findings
- Using only the adversarial loss without the SiD component leads to rapid divergence, even when initialized from SiD.
- Performance is less sensitive to the bias-correction weight alpha in SiDA than in SiD.
Limitations
- SiDA requires access to real training data, unlike data-free SiD.
- Adversarial loss alone diverges, and training can become unstable at large alpha (1.2 or 1.5).
- Classifier-free guidance, two teachers, and post-hoc EMA were not used, so further gains are untested.
Experimental setup
Datasets: CIFAR-10 ImageNet 64x64 FFHQ 64x64 AFHQ-v2 64x64 ImageNet 512x512
Benchmarks: CIFAR-10 ImageNet 64x64 FFHQ 64x64 AFHQ-v2 64x64 ImageNet 512x512
Metrics: FID Inception Score Precision Recall FD_DINOv2
Baselines: VP-EDM SiD DMD Diff-Instruct CTM GDD-I sCD DMD2 BOOT iCT-deep
Builds on: Score identity Distillation (SiD) Diffusion-GAN EDM EDM2
Closest papers by meaning
- Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generation (ICLR 2025 · similarity 0.89)
- Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation (ICML 2024 · similarity 0.87)
- Adversarial Score Distillation: When score distillation meets GAN (CVPR 2024 · similarity 0.87)
- Score Distillation via Reparametrized DDIM (NeurIPS 2024 · similarity 0.86)
- Rethinking Score Distillation as a Bridge Between Image Distributions (NeurIPS 2024 · similarity 0.85)
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.