Research map / Image generation / Fast diffusion sampling
Fast constrained sampling in pre-trained diffusion models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free constrained sampling method for pre-trained latent diffusion models that replaces backpropagation through the denoiser with an approximate Newton step computed via finite-difference Jacobian-vector products (two forward passes).
Problem
Constrained sampling (inpainting, super-resolution, style- and mask-guided generation) with large pre-trained text-to-image diffusion models is slow and memory-hungry when it backpropagates through the denoiser, or fails to propagate constraints over long ranges when it does not.
Approach
Approximate the inverse Jacobian in the Newton step with the denoiser Jacobian itself, so the update direction is J·e, computed via a finite-difference JVP using two forward passes instead of backpropagation or Jacobian inversion (an inexact Newton method).
Key results
- Free-form inpainting: FID 19.39 in 15 seconds versus 32.82 FID in 30 minutes for P2L with captions.
- Box inpainting on ImageNet: FID 42.01 (PSNR 18.30) versus 55.68 for FreeDoM and 74.02 for PSLD, close to fine-tuned SD-Inpaint (FID 32.93).
- Style-guided generation: style score 310.96 with w=5 versus 441.00 for MPGD, at 45 s per image versus 80 s for FreeDoM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use approximate Newton updates for free-form inpainting instead of training-free baselines | Stable Diffusion 1.4 · ImageNet free-form inpainting | FID: 46.72 -> 19.39 | Helped |
| add Qwen-2.5-generated captions | Ours for Stable Diffusion ×8 super-resolution · ImageNet ×8 super-resolution | FID: 60.99 -> 44.74 | Helped |
| add Qwen-2.5-generated captions | Ours for Stable Diffusion ×8 super-resolution · ImageNet ×8 super-resolution | PSNR: 24.26 -> 24.95 | Helped |
| use approximate Newton constrained sampling rather than training-free baselines | Stable Diffusion 1.4 · ImageNet box inpainting | FID: 55.68 -> 42.01 | Helped |
| reduce optimization steps from K=5 to K=2 (ablation) | Ours box inpainting · ImageNet box inpainting | FID: 42.01 -> 68.75 | Hurt |
| reduce optimization steps from K=5 to K=2 (ablation) | Ours box inpainting · ImageNet box inpainting | LPIPS: 0.30 -> 0.39 | Hurt |
| increase learning rate from λ=0.5 to λ=1.0 (ablation) | Ours box inpainting · ImageNet box inpainting | PSNR: 18.30 -> 17.48 | Hurt |
| increase learning rate from λ=0.5 to λ=1.0 (ablation) | Ours box inpainting · ImageNet box inpainting | FID: 42.01 -> 47.20 | Hurt |
| increase style guidance weight from w=2 to w=5 (ablation) | Ours style-guided generation · WikiArt style generation | Style Score: 368.37 -> 310.96 | Helped |
| increase style guidance weight from w=2 to w=5 (ablation) | Ours style-guided generation · WikiArt style generation | CLIP: 23.95 -> 24.57 | Helped |
| use OpenCLIP instead of CLIP for style guidance (ablation) | Ours style-guided generation · WikiArt style generation | Style Score: 310.96 -> 434.45 | Hurt |
| use OpenCLIP instead of CLIP for style guidance (ablation) | Ours style-guided generation · WikiArt style generation | CLIP: 24.57 -> 25.94 | Helped |
Surprising findings
- The denoiser Jacobian is not symmetric, so the approximate Newton direction (J e) differs noticeably from the gradient-descent direction (J^T e).
- Exact forward-mode JVP computation gave no quality gain over the finite-difference approximation while taking 76 s versus 15 s.
Limitations
- High classifier-free guidance (w>10) requires a much smaller learning rate because it alters the denoiser Jacobian spectrum.
- Distilled few-step models need more optimization steps, reducing their speed benefit.
- Non-linear constraints still backpropagate through the VAE decoder and constraint network, and the Newton approximation is applied only to the denoiser.
Experimental setup
Datasets: ImageNet WikiArt PartiPrompts FFHQ MNIST LAION
Benchmarks: ImageNet ctest10k inpainting ImageNet 8x super-resolution ImageNet box inpainting WikiArt/PartiPrompts style transfer FFHQ mask-guided generation
Metrics: PSNR LPIPS FID CLIP-FID CLIP score style score mIoU inference time
Baselines: LDPS PSLD FreeDoM MPGD P2L SD-Inpaint DDIM
Builds on: DDIM classifier-free guidance DPS PSLD FreeDoM
Closest papers by meaning
- Restoration by Generation with Constrained Priors (CVPR 2024 · similarity 0.91)
- CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation (NeurIPS 2025 · similarity 0.90)
- One Step Diffusion via Shortcut Models (ICLR 2025 · similarity 0.90)
- Denoising Diffusion Bridge Models (ICLR 2024 · similarity 0.90)
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.90)
- Constrained Diffusion Models via Dual Training (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.