Research map / Image generation / Fast diffusion sampling
Improved Techniques for Training Consistency Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Improves consistency training (CT) without distillation or LPIPS by removing teacher EMA, using Pseudo-Huber loss, a lognormal noise schedule, an exponential discretization curriculum, and tuned weighting/embeddings/dropout, reaching state-of-the-art one- and two-step FIDs among consistency models on CIFAR-10 and ImageNet 64x64.
Problem
Consistency training (CT) lagged behind consistency distillation (CD), and prior CT relied on the learned LPIPS metric, which may bias FID evaluation and adds compute; the goal is to make CT competitive with or better than CD without learned metrics.
Approach
Identify a flaw in prior theory showing the teacher EMA must be zero for CT, and combine this with Pseudo-Huber metrics, a lognormal noise schedule, doubling discretization curriculum, and tuned weighting/noise embeddings/dropout so CT works with a non-learned metric.
Key results
- iCT-deep achieves one-step FID 2.51 on CIFAR-10 and 3.25 on ImageNet 64x64; two-step FID 2.24 and 2.77.
- Improvements of 3.5x (CIFAR-10) and 4x (ImageNet) in one-step FID over prior CT approaches.
- iCT two-step results surpass CD (LPIPS) in both one-step and two-step settings (CD one-step 3.55 on CIFAR-10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use iCT-deep instead of CT (LPIPS) | Consistency training · CIFAR-10 one-step | FID: 8.70 -> 2.51 | Helped |
| Use iCT-deep instead of CT (LPIPS) | Consistency training · ImageNet 64×64 one-step | FID: 13.0 -> 3.25 | Helped |
| Increase sampling from one step to two steps | iCT-deep · CIFAR-10 | FID: 2.51 -> 2.24 | Helped |
| Increase sampling from one step to two steps | iCT-deep · ImageNet 64×64 | FID: 3.25 -> 2.77 | Helped |
| Remove EMA from the teacher network (ablation) | Consistency training · CIFAR-10 | FID | Helped |
| Use the weighting function 1/(σᵢ₊₁ − σᵢ) instead of uniform weighting (ablation) | Consistency training with squared ℓ2 metric · CIFAR-10 | FID | Helped |
| Reduce the Fourier embedding scale from 16.0 to 0.02 (ablation) | Discrete-time consistency training · CIFAR-10 | FID | Helped |
| Use dropout rate 0.3 instead of zero dropout (ablation) | Consistency training · CIFAR-10 | FID | Helped |
| Replace squared ℓ2 loss with Pseudo-Huber loss (ablation) | Consistency training · CIFAR-10 | FID | Helped |
| Increase the discretization curriculum range from s₀=2, s₁=150 to s₀=10, s₁=1280 (ablation) | Consistency training with Pseudo-Huber loss · CIFAR-10 | FID | Helped |
| Use the exponential curriculum for total discretization steps (ablation) | Consistency training · CIFAR-10 | FID | Helped |
| Use a lognormal noise schedule instead of the original uniform index schedule (ablation) | Consistency training · CIFAR-10 | FID | Helped |
Surprising findings
- Setting the teacher EMA decay to zero (removing EMA) substantially improves CT, contradicting prior theory that EMA was needed.
- Larger-than-diffusion dropout (0.3) helps consistency models despite their single-step nature.
Limitations
- Results rely on tuned hyperparameters (curriculum, dropout, Pseudo-Huber c) evaluated mainly on CIFAR-10 and ImageNet 64x64.
- Only low-resolution image datasets evaluated.
- Remaining gap to leading diffusion models and GANs in some metrics.
Experimental setup
Datasets: CIFAR-10 ImageNet 64x64
Benchmarks: CIFAR-10 ImageNet 64x64
Metrics: FID Inception score precision recall NFE
Baselines: CT (LPIPS) CD (LPIPS) Score SDE DDPM BigGAN-deep StyleGAN2-ADA TRACT PD Diff-Instruct
Builds on: consistency models consistency distillation EDM score SDE
Closest papers by meaning
- Improved Training Technique for Latent Consistency Models (ICLR 2025 · similarity 0.93)
- Sampling is as easy as keeping the consistency: convergence guarantee for Consistency Models (ICML 2024 · similarity 0.91)
- Truncated Consistency Models (ICLR 2025 · similarity 0.91)
- Consistency-GAN: Training GANs with Consistency Model (AAAI 2024 · similarity 0.91)
- Improving Consistency Models with Generator-Augmented Flows (ICML 2025 · similarity 0.91)
- Enhancing Consistency-Based Image Generation via Adversarialy-Trained Classification and Energy-Based Discrimination (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.