Research map / Optimization / Optimizers
Interpreting and Improving Diffusion Models from an Optimization Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Interprets denoising diffusion as approximate gradient descent on the squared distance function to the data manifold, proves DDIM convergence under a relative-error model, and proposes a second-order gradient-estimation sampler reaching state-of-the-art FID in 5-10 steps on CIFAR-10 and CelebA.
Problem
Diffusion sampling is usually described probabilistically, and deterministic samplers like DDIM lack a clear optimization-based explanation or convergence guarantees; fast few-step sampling needs better gradient estimates.
Approach
Interpret the denoiser as estimating the gradient of half the squared distance to the data set (projection), introduce a relative-error model for denoiser output, prove DDIM is approximate gradient descent with admissible geometric noise schedules, and improve gradient estimates by combining current and previous denoiser outputs (second-order update).
Key results
- The proposed gradient-estimation sampler reaches FID 3.79 on CIFAR-10 at N=10 versus 16.86 for DDIM, and 12.57 at N=5 versus 47.20 for DDIM.
- On CelebA, the sampler achieves FID 4.41 at N=10 versus 18.08 for DDIM.
- Ablation at N=10: the schedule alone and the sampler alone each improve over DDIM, and combining both gives FID 3.85 on CIFAR-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DDIM with the gradient-estimation sampler | DDIM · CIFAR-10, N=5, 10, 20, 50 | FID: 47.20 -> 12.57; 16.86 -> 3.79; 8.28 -> 3.32; 4.81 -> 3.41 | Helped |
| replace DDIM with the gradient-estimation sampler | DDIM · CelebA, N=5, 10, 20, 50 | FID: 32.21 -> 10.76; 18.08 -> 4.41; 11.81 -> 3.19; 7.39 -> 3.04 | Helped |
| replace DPM with the gradient-estimation sampler | DPM · CIFAR-10, N=10 | FID: 6.37 -> 3.79 | Helped |
| replace DPM with the gradient-estimation sampler | DPM · CIFAR-10, N=20, 50 | FID: 3.72 -> 3.32; 3.48 -> 3.41 | Mixed |
| replace DPM with the gradient-estimation sampler | DPM · CelebA, N=10 | FID: 5.83 -> 4.41 | Helped |
| replace DPM with the gradient-estimation sampler | DPM · CelebA, N=20, 50 | FID: 2.82 -> 3.19; 2.71 -> 3.04 | Hurt |
| add the improved noise schedule (ablation) | DDIM · CIFAR-10, N=10 | FID: 16.86 -> 13.25 | Helped |
| add the improved noise schedule (ablation) | DDIM · CelebA, N=10 | FID: 18.08 -> 13.55 | Helped |
| add the gradient-estimation sampler improvement (ablation) | DDIM with proposed schedule · CIFAR-10, N=10 | FID: 13.25 -> 3.85 | Helped |
| add the gradient-estimation sampler improvement (ablation) | DDIM with proposed schedule · CelebA, N=10 | FID: 13.55 -> 4.30 | Helped |
| add the proposed noise schedule instead of DDIM schedule (ablation) | DDIM sampler · CIFAR-10, N=10 | FID: 16.86 -> 13.25 | Helped |
| add the proposed noise schedule instead of DDIM schedule (ablation) | DDIM sampler · CelebA, N=10 | FID: 18.08 -> 13.55 | Helped |
Limitations
- The relative-error model is empirically motivated and the theory relies on assumptions (e.g., finite set or manifold, reach) that hold only approximately in practice.
- Final error need not converge to zero under the error model, so the final step is heuristically set to a full step.
- Stable Diffusion results are only comparable to other samplers rather than clearly superior.
Experimental setup
Datasets: CIFAR-10 CelebA MS-COCO
Benchmarks: CIFAR-10 FID CelebA FID MS-COCO 30K FID
Metrics: FID
Baselines: DDIM DDIM Offset EDM PNDM DPM-Solver DEIS UniPC A-DDIM
Builds on: DDIM DDPM EDM Score Distillation Sampling
Closest papers by meaning
- O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions (ICLR 2025 · similarity 0.92)
- Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models (NeurIPS 2024 · similarity 0.92)
- Assessing the quality of denoising diffusion models in Wasserstein distance: noisy score and optimal bounds (NeurIPS 2025 · similarity 0.92)
- AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models (NeurIPS 2025 · similarity 0.90)
- Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training (NeurIPS 2024 · similarity 0.90)
- Denoising Diffusion Bridge Models (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.