Research map / Image generation / Fast diffusion sampling
Score-of-Mixture Training: One-Step Generative Model Training Made Simple via Score Estimation of Mixture Distributions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Score-of-Mixture Training (SMT) and Distillation (SMD), training one-step generators by minimizing alpha-skew Jensen-Shannon divergences via amortized score estimation of real/fake mixture distributions across noise levels.
Problem
Training high-quality one-step generative models stably, without GAN-style discriminator instability, consistency-training sensitivity to noise schedules, or a pretrained diffusion model.
Approach
Minimizes a family of alpha-skew JSD divergences whose generator gradient only needs the score of the mixture of real and fake distributions; this score is estimated with an amortized, alpha- and noise-conditioned denoiser trained by denoising score matching, with an optional GAN-type regularizer; distillation variant parameterizes the mixture score via a pretrained teacher and a fake score model.
Key results
- From scratch on ImageNet 64x64, SMT reaches FID 3.23 with 296M parameters, versus 3.25 for iCT-deep with 592M parameters.
- Distilled SMD reaches FID 1.48 on ImageNet 64x64 in one step, outperforming DMD w/o reg (5.60) and DMD2 w/ GAN (1.51).
- From scratch on CIFAR-10, SMT reaches FID 3.13 in one step, outperforming ECT (3.60) at a similar budget; SMD reaches FID 2.22 on CIFAR-10 distillation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SMT instead of iCT-deep for one-step training from scratch | One-step ImageNet 64x64 generation · ImageNet 64x64 | FID: 3.25 -> 3.23 | Helped |
| Use SMT instead of ECT for one-step training from scratch | One-step ImageNet 64x64 generation · ImageNet 64x64 | FID: 5.51 -> 3.23 | Helped |
| Use SMD instead of DMD2 w/ GAN | One-step ImageNet 64x64 distillation · ImageNet 64x64 | FID: 1.51 -> 1.48 | Helped |
| Use SMD instead of DMD2 (finetuned) | One-step ImageNet 64x64 distillation · ImageNet 64x64 | FID: 1.23 -> 1.48 | Hurt |
| Use SMT instead of ECT for one-step training from scratch | One-step CIFAR-10 generation · CIFAR-10 32x32 | FID: 3.60 -> 3.13 | Helped |
| Use SMD instead of DMD2 w/ GAN | One-step CIFAR-10 distillation · CIFAR-10 32x32 | FID: 2.43 -> 2.22 | Hurt |
| Use SMD instead of SiD | One-step CIFAR-10 distillation · CIFAR-10 32x32 | FID: 1.92 -> 2.22 | Hurt |
| Use the proposed (α,t)-adaptive gradient weighting instead of DMD weighting (ablation) | SMT · CIFAR-10 | FID | Helped |
| Train the score model with a range of α values instead of only α ∈ {0, 1} (ablation) | SMT · CIFAR-10 | FID | Helped |
| Add the GAN-type regularizer (ablation) | SMT · CIFAR-10 | FID | Helped |
Surprising findings
- Learning mixture scores over a range of alpha values outperforms an amortized variant trained only at alpha in {0,1} (the DMD-style reverse-KL case), per the CIFAR-10 ablation.
Limitations
- Only one-step generation is evaluated; few-step variants may improve FID further, as the authors note.
- Evaluation is limited to CIFAR-10 and ImageNet 64x64 image datasets.
- Architecture and training strategy have room for improvement, per the authors.
Experimental setup
Datasets: ImageNet 64x64 CIFAR-10
Benchmarks: ImageNet 64x64 CIFAR-10
Metrics: FID NFE
Baselines: consistency training (CT) iCT iCT-deep ECT DMD DMD2 Diff-Instruct MultiStep-CD CTM SiD
Builds on: consistency models DMD DMD2 EDM Jensen-Shannon divergence GAN
Closest papers by meaning
- Be More Diverse than the Most Diverse: Optimal Mixtures of Generative Models via Mixture-UCB Bandit Algorithms (ICLR 2025 · similarity 0.88)
- Towards Theoretical Understandings of Self-Consuming Generative Models (ICML 2024 · similarity 0.87)
- Analysis of Learning a Flow-based Generative Model from Limited Sample Complexity (ICLR 2024 · similarity 0.87)
- Nearest Neighbour Score Estimators for Diffusion Generative Models (ICML 2024 · similarity 0.87)
- Improved Techniques for Training Consistency Models (ICLR 2024 · similarity 0.86)
- Noise Conditional Variational Score Distillation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.