Research map / Image generation / Fast diffusion sampling
Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STSL, a second-order Tweedie posterior sampler for latent diffusion inverse problems that uses a surrogate loss with a Hessian-trace estimate, reaching strong quality in ~50 steps, and extends it to corruption-robust text-guided image editing.
Problem
Posterior sampling with latent diffusion models for inverse problems is biased by first-order Tweedie approximations, while exact second-order approximations are computationally prohibitive and make reverse diffusion intractable.
Approach
Replace the first-order Tweedie gradient with the gradient of a surrogate loss that includes a Hutchinson-style random-projection estimate of the Hessian trace (from first-order scores only), enabling a tractable second-order correction with O(1) extra compute, plus a DDIM-inversion-based initialization and stochastic averaging.
Key results
- STSL reduces neural function evaluations by 4X vs PSLD and 8X vs P2L, using 50 diffusion steps versus 1000 for baselines, on 8x super-resolution.
- On 8x super-resolution, STSL reaches LPIPS 0.335 on FFHQ vs 0.381 for P2L and 0.402 for PSLD.
- In corrupted-image editing, STSL-CAT attains 93% CLIP accuracy vs 70% for NTI (about 32% relative improvement).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STSL instead of P2L | Stable Diffusion inverse-problem solver · FFHQ-1K, SR (×8) | LPIPS: 0.381 -> 0.335 | Helped |
| Use STSL instead of P2L | Stable Diffusion inverse-problem solver · FFHQ-1K, Motion Deblur | LPIPS: 0.395 -> 0.321 | Helped |
| Use STSL instead of P2L | Stable Diffusion inverse-problem solver · FFHQ-1K, Gaussian Deblur | LPIPS: 0.382 -> 0.308 | Helped |
| Use STSL instead of P2L | Stable Diffusion inverse-problem solver · ImageNet-1K, SR (×8) | LPIPS: 0.441 -> 0.392 | Helped |
| Use STSL instead of P2L | Stable Diffusion inverse-problem solver · ImageNet-1K, Motion Deblur | SSIM: 84.53 -> 76.45 | Hurt |
| Use STSL instead of P2L | Stable Diffusion inverse-problem solver · ImageNet-1K, Gaussian Deblur | LPIPS: 0.402 -> 0.349 | Helped |
| Use STSL instead of PSLD | Stable Diffusion inverse-problem solver · super-resolution 8X inversion | neural function evaluations: 1000 -> 250 | Helped |
| Use STSL instead of P2L | Stable Diffusion inverse-problem solver · super-resolution 8X inversion | neural function evaluations: 2000 -> 250 | Helped |
| Add the second-order trace correction (η = 0.02) (ablation) | STSL · COCO 2017 validation, salt-pepper denoising | LPIPS: 0.404 -> 0.388 | Helped |
| Use 2 stochastic-average samples instead of 1 (ablation) | STSL · COCO 2017 validation, salt-pepper denoising | LPIPS: 0.408 -> 0.403 | Helped |
| Add contrastive loss with ν = 2 instead of ν = 0 (ablation) | STSL · COCO 2017 validation, salt-pepper denoising | LPIPS: 0.404 -> 0.392 | Helped |
| Increase DDIM steps from 5 to 50 (ablation) | STSL · COCO 2017 validation, salt-pepper denoising | LPIPS: 0.564 -> 0.408 | Helped |
Surprising findings
- SSIM is less favorable for STSL than P2L in some cases, which the authors attribute to SSIM rewarding blur-free high-frequency artifacts less and penalizing blur more.
Limitations
- SSIM is lower than P2L in some settings, so perceptual gains do not fully translate to structural metrics.
- P2L runtime is estimated from pseudo-code because source code was unavailable, so efficiency comparisons are approximate.
- Hyperparameters such as the Hessian coefficient must be tuned; too large values change the forward process.
Experimental setup
Datasets: FFHQ ImageNet COCO 2017
Benchmarks: FFHQ-1K ImageNet-1K COCO 2017 val
Metrics: LPIPS PSNR SSIM CLIP accuracy runtime NFE
Baselines: PSLD P2L LDPS GML-DPS LDIR DPS DiffPIR NTI DMPS
Builds on: PSLD P2L DPS DDIM Tweedie's formula
Closest papers by meaning
- Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency (ICLR 2024 · similarity 0.91)
- Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models (ICML 2024 · similarity 0.90)
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.90)
- Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo (ICML 2025 · similarity 0.90)
- Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing (CVPR 2025 · similarity 0.90)
- Fast samplers for Inverse Problems in Iterative Refinement models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.