Research map / Image generation / Fast diffusion sampling
Noise Consistency Training: A Native Approach for One-step Generator in Learning Additional Controls
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Noise Consistency Training (NCT), which adds a ControlNet/IP-Adapter-style adapter to a frozen pre-trained one-step generator and trains it with a noise-space consistency loss plus a boundary loss via primal-dual optimization, enabling controllable single-step generation without retraining or diffusion distillation.
Problem
Adding new control conditions (edges, HED, depth, super-resolution, image prompts) to pre-trained one-step diffusion generators typically requires expensive modification and re-distillation; naive denoising or consistency losses either blur outputs or cannot be applied to a one-step generator that o
Approach
Diffuse the input noise of a frozen one-step generator to multiple noise levels and enforce consistency of the adapted generator's outputs across these partially-coupled noise-condition pairs, anchored by a boundary loss to the frozen generator's outputs on the clean noise, solved as a constrained saddle-point problem with a primal-dual scheme.
Key results
- On the four control tasks averaged (Canny, HED, Depth, 8x super-resolution), NCT reaches avg FID 14.31 and consistency 0.078 at 1 NFE, versus 15.22 / 0.093 for 50-step ControlNet and 15.26 / 0.085 for JDM.
- For image prompting, NCT achieves CLIP-I/CLIP-T comparable to IP-Adapter (100 NFE) with a single forward pass, described as roughly 100x efficiency gain.
- Ablations report that removing the boundary loss causes distribution collapse (FID degradation) and removing the noise consistency loss makes the model ignore the condition (consistency metric degrades severely).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NCT instead of the baselines | One-step controllable generation · Canny | FID: 14.35 -> 13.67 | Helped |
| Use NCT instead of the baselines | One-step controllable generation · Canny | Consistency: 0.122 -> 0.110 | Helped |
| Use NCT instead of the baselines | One-step controllable generation · HED | FID: 16.75 -> 14.96 | Helped |
| Use NCT instead of the baselines | One-step controllable generation · HED | Consistency: 0.055 -> 0.060 | Hurt |
| Use NCT instead of the baselines | One-step controllable generation · Depth | FID: 15.25 -> 16.45 | Mixed |
| Use NCT instead of the baselines | One-step controllable generation · Depth | Consistency: 0.093 -> 0.088 | Helped |
| Use NCT instead of the baselines | One-step controllable generation · 8× Super Resolution | FID: 11.93 -> 12.17 | Mixed |
| Use NCT instead of the baselines | One-step controllable generation · 8× Super Resolution | Consistency: 0.065 -> 0.053 | Helped |
| Replace 50-step ControlNet with NCT | Controllable generation · Canny | NFE: 50 -> 1 | Helped |
| Remove noise consistency loss (ablation) | NCT · Controllable generation | Consistency | Hurt |
| Remove boundary loss (ablation) | NCT · Controllable generation | FID | Hurt |
| Remove primal-dual optimization (ablation) | NCT · Controllable generation | FID | Hurt |
Limitations
- Single-particle MMD estimate used in practice; the paper leaves variance reduction with more particles to future work.
- Experiments are on an internal dataset with limited control types, and the theory is a proof of concept for one-step generators.
Experimental setup
Datasets: internally collected dataset COCO-2017-5k
Benchmarks: COCO-2017-5k
Metrics: FID consistency (L1 between extracted condition and target) CLIP-I CLIP-T NFE
Baselines: ControlNet DI + ControlNet JDM IP-Adapter
Builds on: ControlNet Consistency Models Diff-Instruct IP-Adapter JDM
Closest papers by meaning
- Improving Consistency Models with Generator-Augmented Flows (ICML 2025 · similarity 0.87)
- Denoising Task Difficulty-based Curriculum for Training Diffusion Models (ICLR 2025 · similarity 0.86)
- NoiseCtrl: A Sampling-Algorithm-Agnostic Conditional Generation Method for Diffusion Models (CVPR 2025 · similarity 0.85)
- VCT: Training Consistency Models with Variational Noise Coupling (ICML 2025 · similarity 0.85)
- Noise-free Score Distillation (ICLR 2024 · similarity 0.85)
- Zero-Shot Conditioning of Score-Based Diffusion Models by Neuro-Symbolic Constraints (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.