Research map / Label-efficient and robust learning / Knowledge distillation
Elucidating the Preconditioning in Consistency Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Analytic-Precond, an analytically derived preconditioning for consistency distillation based on a generalized teacher PF-ODE and minimizing a consistency gap, yielding 2x-3x training acceleration for consistency trajectory models in multi-step generation.
Problem
Preconditioning in consistency distillation (CMs and CTMs) is hand-crafted and possibly suboptimal; its design criteria are not well understood, and CTMs have not found a preconditioning that outperforms the original.
Approach
Relate preconditioning coefficients to discretization of the teacher PF-ODE, generalize the ODE with free parameters (l_t, s_t) that preserve boundary conditions, and analytically choose these parameters to minimize a bound on the consistency gap between teacher and optimal student denoisers.
Key results
- On CTM with two-step generation, Analytic-Precond gives up to 3x training acceleration over CTM's preconditioning across datasets.
- Table of multi-step FID: CIFAR-10 unconditional NFE=2 FID 3.77 vs CTM 3.83; CIFAR-10 conditional NFE=2 2.92 vs 3.00; FFHQ NFE=10 5.12 vs 5.23.
- For CMs and single-step CTMs, Analytic-Precond yields training curves similar to the original preconditioning, since coefficients coincide when jumping to epsilon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace CTM preconditioning with Analytic-Precond | CTM · FFHQ 64×64 unconditional, NFE 2–10 | FID: 5.96 -> 5.71; 5.80 -> 5.56; 5.53 -> 5.47; 5.39 -> 5.31; 5.23 -> 5.12 | Helped |
| Replace CTM preconditioning with Analytic-Precond | CTM · ImageNet 256×256 conditional, NFE 2–10 | FID: 5.95 -> 5.73; 6.16 -> 5.67; 5.43 -> 5.34; 5.44 -> 5.43; 5.98 -> 5.70 | Helped |
| Replace CTM preconditioning with Analytic-Precond | CTM · CIFAR-10 unconditional, NFE 2–10 | FID: 3.83 -> 3.77; 3.58 -> 3.54; 3.43 -> 3.38; 3.33 -> 3.30; 3.22 -> 3.25 | Mixed |
| Replace CTM preconditioning with Analytic-Precond | CTM · CIFAR-10 conditional, NFE 2–10 | FID: 3.00 -> 2.92; 2.82 -> 2.75; 2.59 -> 2.62; 2.67 -> 2.60; 2.56 -> 2.65 | Mixed |
| Use Analytic-Precond instead of the original CTM preconditioning for intermediate jumps | CTM multi-step training · CIFAR-10, FFHQ 64×64, and ImageNet 64×64 | training acceleration: up to 3× | Helped |
Surprising findings
- For CMs and single-step CTMs, the analytically optimized preconditioning gives comparable performance to the original, indicating the original is already near-optimal for jumps to the initial timestep.
- Replacing CTM's preconditioning with BCM's does not improve one-step or two-step generation.
Limitations
- Final FID improvement is relatively insignificant despite training acceleration.
- No improvement over prior preconditioning on long jumps, so single-step performance is comparable.
- With GAN loss, the consistency function no longer faithfully follows the teacher trajectory, weakening the theoretical basis.
Experimental setup
Datasets: CIFAR-10 FFHQ 64x64 ImageNet 64x64
Benchmarks: CIFAR-10 FFHQ 64x64 ImageNet 64x64
Metrics: FID MSE
Baselines: CM CTM BCM
Builds on: consistency models consistency trajectory models EDM Rosenbrock-type exponential integrators
Closest papers by meaning
- Improving Consistency Models with Generator-Augmented Flows (ICML 2025 · similarity 0.91)
- Truncated Consistency Models (ICLR 2025 · similarity 0.89)
- Consistency Models Made Easy (ICLR 2025 · similarity 0.88)
- Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion (ICLR 2024 · similarity 0.88)
- How to build a consistency model: Learning flow maps via self-distillation (NeurIPS 2025 · similarity 0.88)
- SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.