Research map / Image generation / Fast diffusion sampling
NoiseCtrl: A Sampling-Algorithm-Agnostic Conditional Generation Method for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NoiseCtrl, a training-free conditional generation method that replaces isotropic Gaussian sampling noise in diffusion models with directionally biased noise (von Mises-Fisher approximation), leaving the sampler unchanged.
Problem
Training-free conditional diffusion generation usually steers the reverse process with differentiable loss gradients, which requires modifying sampling algorithms (DDPM, DDIM, etc.) and reduces their stability and flexibility.
Approach
Instead of altering the sampler, replace the isotropic Gaussian noise at each step with conditional noise whose direction is optimized to reduce a condition energy loss on the predicted clean sample; the direction is sampled from a von Mises-Fisher distribution (kappa=1e4) and a fast closed-form spherical gradient estimate (as in DSG) gives the direction.
Key results
- On depth conditions, DDPM fast,2 achieves FID 22.76 versus 26.11 for UGD, 27.34 for FreeDoM and 27.25 for DSG.
- Across the six NoiseCtrl variants, DDPM/DDIM/LCM/DPM/Euler/SA fast,2 are reported to be nearly identical in runtime to their base samplers (e.g., DDPM 6 s vs 16 s NoiseCtrl fast,2), while UGD takes 4507 s.
- Depth-condition MSE drops to about 86-88 for NoiseCtrl variants versus 95-97 for training-free baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace isotropic Gaussian noise with conditional noise sampled from the von Mises–Fisher approximation | Stable Diffusion v1.5 with NoiseCtrl · Depth | FID: 27.1109 -> 22.7612 | Helped |
| Replace isotropic Gaussian noise with conditional noise sampled from the von Mises–Fisher approximation | Stable Diffusion v1.5 with NoiseCtrl · Canny | FID: 26.3717 -> 21.2472 | Helped |
| Replace isotropic Gaussian noise with conditional noise sampled from the von Mises–Fisher approximation | Stable Diffusion v1.5 with NoiseCtrl · Style | FID: 26.0756 -> 26.3752 | Hurt |
| Use fast rather than slow direction-vector estimation (ablation) | NoiseCtrl · Conditional generation | inference time: DDPMslow,2 54 -> DDPMfast,2 16 | Helped |
| Sample conditional noise from the von Mises–Fisher approximation (distribution 2) rather than piecewise Gaussian distribution (distribution 1) (ablation) | NoiseCtrl · Conditional generation | inference time: DDPMfast,1 23 -> DDPMfast,2 16 | Helped |
| Increase the von Mises–Fisher concentration parameter κ (ablation) | NoiseCtrl · Conditional image generation | MSE controllability | Helped |
| Use a smaller von Mises–Fisher concentration parameter κ (ablation) | NoiseCtrl · Conditional image generation | randomness | Helped |
Surprising findings
- Reported E[d^T u] = kappa/(kappa+n-1) implies controllability increases with kappa, but very small kappa produces random images rather than control.
- The slow and fast direction estimation variants yield nearly identical results according to the authors, so only the fast version is shown.
Limitations
- Relies on the same closed-form spherical gradient as DSG and on a differentiable energy/loss for each condition type.
- Slow direction estimation is computationally expensive and the piecewise-Gaussian distribution is approximated with von Mises-Fisher for efficiency.
- Hyperparameter kappa must be tuned: too small gives randomness, too large limits diversity; quantitative evaluation limited to 5000 images per condition.
Experimental setup
Datasets: COCO2017 validation set WikiArt
Benchmarks: COCO2017 val (depth, canny, style conditions) WikiArt style images
Metrics: FID CLIP-score CLIP aesthetic score MSE controllability style loss inference time
Baselines: UGD FreeDom DSG ControlNet T2I-Adapter IP-Adapter
Builds on: DSG (Guidance with Spherical Gaussian Constraint) DDPM DDIM FreeDoM Universal Guided Diffusion
Closest papers by meaning
- A Simple Approach to Unifying Diffusion-based Conditional Generation (ICLR 2025 · similarity 0.91)
- CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation (NeurIPS 2025 · similarity 0.90)
- DEFT: Efficient Fine-tuning of Diffusion Models by Learning the Generalised $h$-transform (NeurIPS 2024 · similarity 0.89)
- Compressed Image Generation with Denoising Diffusion Codebook Models (ICML 2025 · similarity 0.89)
- Rectified Diffusion Guidance for Conditional Generation (CVPR 2025 · similarity 0.89)
- One Step Diffusion via Shortcut Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.