Research map / Image generation / Fast diffusion sampling
AdjointDPM: Adjoint Sensitivity Method for Gradient Backpropagation of Diffusion Probabilistic Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdjointDPM, which backpropagates losses through diffusion sampling to initial noises, conditioning embeddings, and UNet weights by solving an adjoint ODE after reparameterizing the probability-flow ODE with exponential integration, giving constant memory.
Problem
Optimizing parameters of diffusion probabilistic models (noise states, conditioning text embeddings, network weights) against differentiable losses on generated outputs requires backpropagating through all sampling iterations, which is memory-intensive.
Approach
Apply the adjoint sensitivity method from neural ODEs to the probability-flow ODE sampling process, and reparameterize the forward and adjoint ODEs with exponential integration (semi-linear structure) into non-stiff ODEs so that discretization error affects only the nonlinear part; memory stays constant in the number of steps.
Key results
- On CIFAR-10 at NFE 10, the reparameterized adjoint ODE sampler reaches FID 4.36 versus 9.50 for solving the original ODE, close to DPM-solver's 4.70.
- On Stanford Dogs, AdjointDPM gives a -7.1% per-class FID change versus -5.3% for DOODL.
- Stylization CLIP similarity on CIFAR-100 prompts is higher for AdjointDPM (e.g., 27.34 on Airplane) than DreamBooth and Textual Inversion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reparameterize the forward probability-flow ODE using exponential integration | VPSDE sampling · CIFAR10, NFE 10 | FID: 9.50 -> 4.36 | Helped |
| Reparameterize the forward probability-flow ODE using exponential integration | VPSDE sampling · CIFAR10, NFE 20 | FID: 8.27 -> 2.90 | Helped |
| Reparameterize the forward probability-flow ODE using exponential integration | VPSDE sampling · CIFAR10, NFE 50 | FID: 5.64 -> 2.58 | Helped |
| Use reparameterized ODE sampling instead of DPM-solver | VPSDE sampling · CIFAR10, NFE 10 | FID: 4.70 -> 4.36 | Helped |
| Use reparameterized ODE sampling instead of DPM-solver | VPSDE sampling · CIFAR10, NFE 20 | FID: 2.87 -> 2.90 | Hurt |
| Use reparameterized ODE sampling instead of DPM-solver | VPSDE sampling · CIFAR10, NFE 50 | FID: 2.62 -> 2.58 | Helped |
| Guide Stable Diffusion sampling with a fine-grained dog classifier using AdjointDPM | Stable Diffusion · Stanford Dogs | Per-class % FID change: -5.3% -> -7.1% | Helped |
| Optimize initial noise to mislead the ImageNet classifier | ImageNet diffusion model · ImageNet, 10 randomly chosen classes | Attack rate: 51.2% | Helped |
| Fine-tune Stable Diffusion with AdjointDPM for stylization | Stable Diffusion · Stylization prompts: Airplane, Clock, House, Cat, Apples | CLIP similarity: 21.98 -> 27.34; 28.23 -> 30.00; 25.40 -> 29.03; 25.90 -> 26.83; 28.10 -> 28.59 | Helped |
| Fine-tune Stable Diffusion with AdjointDPM for stylization | Stable Diffusion · Stylization prompts: Airplane, Clock, House, Cat, Apples | CLIP similarity: 24.44 -> 27.34; 26.74 -> 30.00; 25.91 -> 29.03; 22.30 -> 26.83; 24.06 -> 28.59 | Helped |
Surprising findings
- Adversarial initial noises within a small perturbation bound can bypass the Stable Diffusion NSFW filter, raising security concerns.
- Backpropagating to one initial noise used 19.63GB versus 23.24GB for DOODL on a V100, a modest memory difference.
Limitations
- Memory comparison is limited to one setting against DOODL, and time comparisons are qualitative.
- The adversarial success ratio is only reported on a subset of ImageNet classes.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet Stanford Dogs LAION
Benchmarks: CIFAR-10 FID Stanford Dogs FID CIFAR-100 stylization CLIP similarity
Metrics: FID CLIP similarity attack success ratio peak memory (GB)
Baselines: DOODL DreamBooth Textual Inversion FlowGrad DPM-solver
Builds on: neural ODE adjoint method DPM-solver classifier-free guidance DOODL DreamBooth
Closest papers by meaning
- AdjointDEIS: Efficient Gradients for Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Efficient Adjoint Matching for Fine-tuning Diffusion Models (NeurIPS 2026 · similarity 0.88)
- Adjoint Sampling: Highly Scalable Diffusion Samplers via Adjoint Matching (ICML 2025 · similarity 0.87)
- UDPM: Upsampling Diffusion Probabilistic Models (NeurIPS 2024 · similarity 0.86)
- On Inference Stability for Diffusion Models (AAAI 2024 · similarity 0.86)
- Enhanced Diffusion Sampling via Extrapolation with Multiple ODE Solutions (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.