Research map / Image generation / Fast diffusion sampling
Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReSample, which solves general (linear and nonlinear) inverse problems with pretrained latent diffusion models by enforcing hard data consistency via optimization in latent space and remapping the result back to the noisy latent manifold with stochastic resampling.
Problem
Solving linear and nonlinear inverse problems with latent diffusion models is hard because the encoder and decoder are nonlinear, so gradient-based data consistency in latent space gets stuck in poor local minima and produces noisy or artifact-laden reconstructions.
Approach
Enforce measurement consistency as a hard constrained optimization over the latent (starting from the Tweedie estimate) at selected diffusion steps, then map the consistent sample back to noisy latent at time t via a stochastic resampling posterior that also conditions on the unconditional sample, reducing variance versus stochastic encoding.
Key results
- On CelebA-HQ super-resolution 4x, ReSample reaches LPIPS 0.144, PSNR 30.45, SSIM 0.832, beating DPS (28.41 PSNR) and PSLD-LDM (27.61 PSNR).
- On CelebA-HQ nonlinear deblurring, ReSample achieves PSNR 30.18 vs. DPS 26.81.
- On LDCT chest/abdomen/head CT reconstruction, ReSample achieves the best PSNR/SSIM across all three anatomies (e.g., abdominal 35.91 dB vs. 32.84 for PnP-UNet).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReSample instead of the strongest reported baseline | CelebA-HQ super-resolution 4× · CelebA-HQ super-resolution 4× | PSNR: 29.49 ± 1.93 -> 30.45 ± 2.09 | Helped |
| Use ReSample instead of DPS | CelebA-HQ random inpainting 70% · CelebA-HQ random inpainting 70% | PSNR: 32.48 ± 2.30 -> 32.77 ± 2.23 | Helped |
| Use ReSample instead of DPS | CelebA-HQ nonlinear deblurring · CelebA-HQ nonlinear deblurring | PSNR: 26.81 ± 2.84 -> 30.18 ± 2.21 | Helped |
| Use ReSample instead of DPS | CelebA-HQ Gaussian deblurring · CelebA-HQ Gaussian deblurring | PSNR: 28.36 ± 2.12 -> 30.69 ± 2.14 | Helped |
| Use ReSample instead of the strongest reported baseline | LDCT CT reconstruction · Abdominal, head, and chest CT | PSNR: 32.84 ± 1.29 -> 35.91 ± 1.22; 33.45 ± 3.25 -> 37.82 ± 5.31; 29.78 ± 1.12 -> 31.72 ± 0.912 | Helped |
| Use ReSample instead of DPS | FFHQ super-resolution 4× · FFHQ super-resolution 4× | PSNR: 28.47 ± 2.09 -> 28.90 ± 1.86 | Helped |
| Use ReSample instead of DPS | FFHQ random inpainting 70% · FFHQ random inpainting 70% | PSNR: 32.32 ± 2.21 -> 31.34 ± 2.11 | Hurt |
| Use ReSample instead of DPS | FFHQ nonlinear deblurring · FFHQ nonlinear deblurring | PSNR: 27.13 ± 3.11 -> 30.03 ± 1.78 | Helped |
| Use ReSample instead of DPS | FFHQ Gaussian deblurring · FFHQ Gaussian deblurring | PSNR: 27.70 ± 2.04 -> 28.73 ± 1.87 | Helped |
| Use ReSample instead of PSLD-LDM | FFHQ 1k box inpainting · FFHQ 1k box inpainting | PSNR: 24.22 -> 23.81 | Hurt |
| Replace stochastic encoding with stochastic resampling (ablation) | ReSample · LSUN-Bedroom and CelebA-HQ Gaussian deblurring and super-resolution | PSNR | Helped |
| Increase hard data consistency frequency (ablation) | ReSample · CT reconstruction | PSNR | Helped |
Surprising findings
- Authors found that PSLD results were underreported; after re-tuning gamma with the original authors, PSLD baseline numbers changed substantially.
- Latent-DPS gradient added on top of ReSample gives only marginal PSNR gains and only with carefully scaled learning rate.
Limitations
- Hard data consistency adds computational overhead compared with methods that apply consistency at every step.
- Performance is sensitive to resampling hyperparameter gamma and skip step size; the approach is evaluated mainly on 256x256 images with a 64x64 latent autoencoder.
- Gains over DPS are smaller on linear tasks like random inpainting, and on FFHQ ReSample is comparable to DPS for Gaussian deblurring.
Experimental setup
Datasets: FFHQ CelebA-HQ LSUN-Bedroom AAPM LDCT
Benchmarks: CelebA-HQ FFHQ AAPM LDCT
Metrics: LPIPS PSNR SSIM
Baselines: DPS MCG DDRM DMPS ADMM-PnP Latent-DPS PSLD PnP-UNet FBP-UNet FBP
Builds on: Diffusion Posterior Sampling (DPS) Latent Diffusion Models DDIM Tweedie's formula
Closest papers by meaning
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.91)
- Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models (ICML 2024 · similarity 0.91)
- SITCOM: Step-wise Triple-Consistent Diffusion Sampling For Inverse Problems (ICML 2025 · similarity 0.91)
- Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion (CVPR 2024 · similarity 0.91)
- Prompt-tuning Latent Diffusion Models for Inverse Problems (ICML 2024 · similarity 0.90)
- Fast samplers for Inverse Problems in Iterative Refinement models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.