Research map / Image generation / Autoregressive and latent generation
Improved Training Technique for Latent Consistency Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts improved consistency training (iCT) to latent space by replacing Pseudo-Huber with Cauchy loss, adding early-timestep diffusion loss, OT coupling, an adaptive scaling-c schedule, and Non-scaling LayerNorm, yielding 1-2 step latent consistency models on 256x256 faces and churches.
Problem
Consistency training performs poorly in latent space because latent data contains highly impulsive outliers that destabilize the TD-based consistency loss, unlike bounded pixel data.
Approach
Identify impulsive outliers in latent TD values as the cause of iCT failure in latent space, and fix it with a Cauchy loss, adaptive robustness scaling c, and Non-scaling LayerNorm, plus a small-timestep diffusion loss and OT minibatch coupling.
Key results
- On CelebA-HQ 256x256 with 1-NFE, the method reaches FID 7.27 versus 37.15 for iLCT (baseline iCT in latent space).
- With 2-NFE, FID on CelebA-HQ is 6.93, and FIDs of 8.87 (LSUN Church, 1-NFE) and 8.72 (FFHQ, 1-NFE) are reported.
- The 1-NFE model outperforms the same-architecture LDM-8 trained with the same VE noise (FID 8.85 on CelebA-HQ), but remains behind LDM-4 (FID 5.11) and LFM-8 (FID 5.82).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Pseudo-Huber with Cauchy loss (ablation) | iLCT on CelebA-HQ · CelebA-HQ 256×256, 1 NFE | FID: 37.15 -> 13.02 | Helped |
| add diffusion loss at small timesteps (ablation) | Cauchy latent consistency model · CelebA-HQ 256×256, 1 NFE | FID: 13.02 -> 9.11 | Helped |
| add OT coupling during minibatch training (ablation) | Cauchy model with diffusion loss · CelebA-HQ 256×256, 1 NFE | FID: 9.11 -> 8.89 | Helped |
| replace fixed c with adaptive scaling-c schedule (ablation) | model with OT coupling · CelebA-HQ 256×256, 1 NFE | FID: 8.89 -> 7.76 | Helped |
| use Non-scaling LayerNorm (ablation) | model with adaptive c schedule · CelebA-HQ 256×256, 1 NFE | FID: 7.76 -> 7.27 | Helped |
| apply diffusion loss at every timestep (r=1.0) instead of selectively at r=0.25 (ablation) | proposed latent consistency model · CelebA-HQ 256×256, 1 NFE | FID: 7.27 -> 7.47 | Hurt |
| use diffusion loss threshold r=0.6 instead of r=0.25 (ablation) | proposed latent consistency model · CelebA-HQ 256×256, 1 NFE | FID: 7.27 -> 7.33 | Hurt |
| replace Cauchy with L2 loss (ablation) | proposed latent consistency model · CelebA-HQ 256×256, 1 NFE | FID: 7.27 -> 50.40 | Hurt |
| replace Cauchy with E-LatentLPIPS (ablation) | proposed latent consistency model · CelebA-HQ 256×256, 1 NFE | FID: 7.27 -> 11.49 | Hurt |
| replace Cauchy with Huber loss (ablation) | proposed latent consistency model · CelebA-HQ 256×256, 1 NFE | FID: 7.27 -> 9.97 | Hurt |
| replace Cauchy with Geman-McClure loss (ablation) | proposed latent consistency model · CelebA-HQ 256×256, 1 NFE | FID: 7.27 -> 11.28 | Hurt |
| use GroupNorm instead of Non-scaling LayerNorm (ablation) | proposed latent consistency model · CelebA-HQ 256×256, 1 NFE | FID: 7.27 -> 7.76 | Hurt |
Surprising findings
- Geman-McClure and Huber losses underperform Cauchy in latent space, as Geman-McClure ignores impulsive outliers entirely.
- Normalization ablation shows GroupNorm and InstanceNorm outperform standard LayerNorm; removing LayerNorm's scaling term (NsLN) is what makes it competitive.
Limitations
- 1-NFE and 2-NFE FIDs still lag behind multi-step latent diffusion models such as LDM-4 and LFM-8.
- Adaptive c scheduler is a heuristic fitted from empirical observations.
- Experiments are limited to 256x256 face and church datasets; no text-to-image or video results.
Experimental setup
Datasets: CelebA-HQ LSUN Church FFHQ CIFAR-10
Benchmarks: CelebA-HQ 256x256 LSUN Church 256x256 FFHQ 256x256
Metrics: FID Recall NFE
Baselines: iLCT LDM-8 LDM-4 LFM-8 WaveDiff DDGAN RDUOT Score SDE DIMSUM LSGM RDM E-LatentLPIPS
Builds on: consistency models improved consistency training (iCT) EDM latent diffusion models OpenAI UNet
Closest papers by meaning
- Improved Techniques for Training Consistency Models (ICLR 2024 · similarity 0.93)
- Truncated Consistency Models (ICLR 2025 · similarity 0.90)
- Phased Consistency Models (NeurIPS 2024 · similarity 0.90)
- Simplifying, Stabilizing and Scaling Continuous-time Consistency Models (ICLR 2025 · similarity 0.90)
- Sampling is as easy as keeping the consistency: convergence guarantee for Consistency Models (ICML 2024 · similarity 0.89)
- Theory of Consistency Diffusion Models: Distribution Estimation Meets Fast Sampling (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.