Research map / Image generation / Fast diffusion sampling
Universal Guidance for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes universal guidance, a sampler that steers frozen diffusion models with arbitrary off-the-shelf differentiable guidance functions (CLIP, segmentation, face recognition, object detection) by evaluating guidance on predicted clean images, plus backward guidance and per-step self-recurrence.
Problem
Diffusion models are typically trained for a single conditioning modality and need retraining to accept others; standard guidance with off-the-shelf networks fails because those networks are trained on clean images and receive noisy inputs.
Approach
Compute guidance gradients on the predicted clean image rather than the noisy latent (forward guidance), add a backward guidance step that optimizes a clean-space change and maps it back to the noise prediction, and apply per-step self-recurrence to keep images realistic, all without retraining the diffusion model or guidance network.
Key results
- Universal guidance with Stable Diffusion and CLIP guidance produces images comparable to text-conditional Stable Diffusion for matching prompts (qualitative comparison only).
- Forward plus backward guidance yields realistic ImageNet diffusion images with objects in designated bounding-box locations, whereas forward-only guidance often yields wrong categories or positions (qualitative).
- Segmentation-guided inpainting combines inpainting, classifier and segmentation guidance on the ImageNet model, producing realistic images matching both the mask and class label.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use universal CLIP guidance instead of text conditioning | Stable Diffusion · Text-prompt image generation | image quality and prompt match | No clear effect |
| Add backward universal guidance to forward guidance (ablation) | ImageNet diffusion with object-location guidance · Object-location-guided image generation | object category and location match | Helped |
| Add segmentation guidance to classifier-guided inpainting (ablation) | ImageNet diffusion · Segmentation-guided inpainting | segmentation and inpainting match | Helped |
Surprising findings
- Simply increasing guidance strength is suboptimal, often causing instability as images leave the data manifold faster than the denoiser can correct.
Limitations
- Generation is substantially slower than conditional sampling, scaling linearly with recurrence steps k.
- Backward guidance requires an inner multi-step gradient descent loop that is expensive for large guidance networks.
- Optimal sampling hyperparameters (guidance strength, k) must be tuned individually for each guidance network.
Experimental setup
Datasets: ImageNet
Baselines: text-conditional Stable Diffusion classifier guidance unguided generation
Builds on: classifier guidance classifier-free guidance DDIM Stable Diffusion CLIP
Closest papers by meaning
- Understanding and Improving Training-free Loss-based Diffusion Guidance (NeurIPS 2024 · similarity 0.89)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.89)
- REG: Rectified Gradient Guidance for Conditional Diffusion Models (ICML 2025 · similarity 0.88)
- Elucidating the design space of classifier-guided diffusion generation (ICLR 2024 · similarity 0.88)
- Theoretical insights for diffusion guidance: A case study for Gaussian mixture models (ICML 2024 · similarity 0.88)
- Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.