Research map / Image generation / Fast diffusion sampling
Adaptive Discretization for Consistency Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADCMs, which adaptively choose the discretization step for consistency training by balancing local consistency (objective) against global denoising error (constraint) via a Lagrange multiplier and a Gauss-Newton closed-form step, improving CM training efficiency on CIFAR-10 and ImageNet 64x64.
Problem
Choosing the discretization step in consistency model training is manual and dataset/noise-schedule dependent, and poor choices hurt trainability and stability.
Approach
Formulates discretization as a constrained optimization: minimize local consistency loss subject to a bound on global denoising error, relaxes it with a Lagrange multiplier, and solves for the step analytically with a first-order Taylor (JVP) Gauss-Newton approximation; also adds an adaptive weighting (1/global error) and Pseudo-Huber metric.
Key results
- On unconditional CIFAR-10, ADCM reaches 1-step FID 3.16 with 12.8M training images, versus ECM's 3.60 at 51.2M images.
- ADCM achieves 1-step FID 2.80 on CIFAR-10 with 76.8M images, versus iCT's 2.83 at 409.6M images.
- On class-conditional ImageNet 64x64 with 2x model size, ADCM achieves 1-step FID 3.04 at 51.2M images, and 3.49 at 12.8M images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive discretization instead of the ECM discretization | consistency training on CIFAR-10 · CIFAR-10, 12.8M images | 1-step FID: 4.54 -> 3.16 | Helped |
| use adaptive discretization instead of ECM at equal budget and model size | consistency training on ImageNet 64x64 · ImageNet 64x64, 12.8M images, 1x model | 1-step FID: 5.51 -> 5.12 | Helped |
| increase ADCM model size from 1x to 2x | ADCM · ImageNet 64x64, 12.8M images | 1-step FID: 5.12 -> 3.49 | Helped |
| increase ADCM training budget from 12.8M to 51.2M images | ADCM · ImageNet 64x64, 2x model | 1-step FID: 3.49 -> 3.04 | Helped |
| use adaptive discretization instead of ECM | consistency training on ImageNet 512x512 · ImageNet 512x512, 6.4M images, 1x model | 1-step FID: 25.69 -> 23.12 | Helped |
| use adaptive discretization instead of ECM | consistency training on Flow Matching · Flow Matching | FID: 5.82 -> 5.14 | Helped |
| add adaptive weighting instead of constant weighting (ablation) | ADCM · CIFAR-10, 12.8M images | 1-step FID: 5.70 -> 3.16 | Helped |
| add adaptive weighting instead of inverse-interval weighting (ablation) | ADCM · CIFAR-10, 12.8M images | 1-step FID: 4.09 -> 3.16 | Helped |
| add adaptive weighting instead of inverse-time weighting (ablation) | ADCM · CIFAR-10, 12.8M images | 1-step FID: 3.84 -> 3.16 | Helped |
| use Pseudo-Huber c=0.03 instead of L2 distance (ablation) | ADCM · CIFAR-10, 12.8M images | 1-step FID: 3.54 -> 3.16 | Helped |
| set Pseudo-Huber c to 0.0003 instead of 0.03 (ablation) | ADCM · CIFAR-10, 12.8M images | 1-step FID: 3.16 -> 3.44 | Hurt |
| set Pseudo-Huber c to 0.003 instead of 0.03 (ablation) | ADCM · CIFAR-10, 12.8M images | 1-step FID: 3.16 -> 3.42 | Hurt |
Surprising findings
- A large Lagrange multiplier improves stability but hurts final performance, while too small a multiplier reduces stability (trade-off).
- Without adjustments, baseline ECM's 1-step FID on Flow Matching degrades from 5.82 to 15.55 when its discretization is not manually modified.
Limitations
- Focuses on Consistency Training; for Consistency Distillation, estimating the global loss is costly due to iterative PF-ODE solving.
- No error bars reported; standard deviations are claimed small based on 50k-sample FID.
- Relies on first-order Taylor and Gauss-Newton approximations; requires a warm-up Lagrange multiplier on ImageNet for stability.
Experimental setup
Datasets: CIFAR-10 ImageNet 64x64 ImageNet 512x512
Benchmarks: CIFAR-10 ImageNet 64x64 ImageNet 512x512
Metrics: FID NFE
Baselines: iCT ECM sCT sCM CD (LPIPS) CCM TCM CTM
Builds on: Consistency Models Consistency Training (iCT) EDM Pseudo-Huber metric sCM
Closest papers by meaning
- Improved Techniques for Training Consistency Models (ICLR 2024 · similarity 0.90)
- Simplifying, Stabilizing and Scaling Continuous-time Consistency Models (ICLR 2025 · similarity 0.89)
- Truncated Consistency Models (ICLR 2025 · similarity 0.89)
- VCT: Training Consistency Models with Variational Noise Coupling (ICML 2025 · similarity 0.88)
- Improved Training Technique for Latent Consistency Models (ICLR 2025 · similarity 0.88)
- Consistency Models Made Easy (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.