Research map / Image generation / Fast diffusion sampling
Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Flash-Diffusion, which predicts degradation severity of corrupted images in an autoencoder latent space (severity encoding) and uses it to set per-sample start times in latent diffusion sampling, wrapping existing latent diffusion inverse-problem solvers for sample-adaptive, up to 10x faster reconstruction.
Problem
Diffusion-based inverse problem solvers spend the same number of function evaluations on every sample, although reconstruction difficulty varies strongly with signal structure, degradation severity, and their interaction.
Approach
A severity encoder fine-tuned from a pretrained LDM encoder predicts the latent estimate and its error variance; the variance is mapped via SNR matching to a per-sample starting time in the latent reverse diffusion process, so the number of sampling steps adapts to each sample, and the framework wraps any latent diffusion solver.
Key results
- Adaptive Latent-DPS, PSLD, GML-DPS and ReSample consistently outperform their baselines on all degradations and both datasets, e.g. FFHQ varying Gaussian deblurring Latent-DPS FID 87.26 to 55.57 and LPIPS 0.3579 to 0.2240.
- Flash-Diffusion achieves roughly 8x-10x reduction in average number of function evaluations (e.g. Latent-DPS 1000 to 100.3 NFE on FFHQ varying blur).
- SwinIR attains the highest PSNR/SSIM (e.g. 30.69 dB on FFHQ varying blur) but worse LPIPS and FID than the adapted diffusion solvers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add sample-adaptive severity-based sampling to Latent-DPS | Latent-DPS · FFHQ and LSUN Bedrooms, all four degradations | LPIPS: FFHQ varying Gaussian blur: 0.3579 -> 0.2240; LSUN varying Gaussian blur: 0.4227 -> 0.2185 | Helped |
| Add sample-adaptive severity-based sampling to ReSample | ReSample · FFHQ and LSUN Bedrooms, all four degradations | LPIPS: FFHQ varying Gaussian blur: 0.2587 -> 0.2383; LSUN varying Gaussian blur: 0.2499 -> 0.2284 | Helped |
| Add sample-adaptive severity-based sampling | Latent-DPS · FFHQ and LSUN Bedrooms, all four degradations | mean reverse diffusion steps: 1000 -> 100.3; 1000 -> 125.3; 1000 -> 136.1; 1000 -> 104.7 (FFHQ) | Helped |
| Enable sample-by-sample adaptation rather than fixed start time (ablation) | Latent-DPS, FFHQ Gaussian deblurring · Gaussian deblurring | LPIPS: 0.2346 -> 0.2241 | Helped |
| Enable sample-by-sample adaptation rather than fixed start time (ablation) | Latent-DPS, FFHQ nonlinear deblurring · Nonlinear deblurring | LPIPS: 0.2847 -> 0.2694 | Helped |
| Use latent-space rather than pixel-space diffusion (ablation) | Fixed-start Latent-DPS, FFHQ Gaussian deblurring · Gaussian deblurring | LPIPS: 0.2377 -> 0.2346 | Helped |
| Use latent-space rather than pixel-space diffusion (ablation) | Fixed-start Latent-DPS, FFHQ nonlinear deblurring · Nonlinear deblurring | LPIPS: 0.2843 -> 0.2847 | Hurt |
Surprising findings
- Adaptive Latent-DPS often outperforms more advanced baselines such as ReSample once wrapped, narrowing the gap between solvers.
- The severity encoder's predictions break down at test-time noise levels well above the training noise (sigma 0.1 vs 0.05 training).
Limitations
- Severity encoder must be trained in a supervised way on the specific degraded image distribution and forward model.
- Severity estimates break down under large test-time noise mismatch and forward model mismatch.
- Baseline gains depend on a sample-adaptive start time, and the method may hallucinate details as with any generative reconstruction.
Experimental setup
Datasets: CelebA-HQ LSUN Bedrooms FFHQ
Benchmarks: FFHQ LSUN Bedrooms
Metrics: PSNR SSIM LPIPS FID average NFE
Baselines: Latent-DPS GML-DPS PSLD ReSample DPS SwinIR
Builds on: Latent Diffusion Models Diffusion Posterior Sampling Come-Closer-Diffuse-Faster ReSample PSLD
Closest papers by meaning
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.92)
- Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency (ICLR 2024 · similarity 0.91)
- InvFusion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems (NeurIPS 2025 · similarity 0.91)
- Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing (CVPR 2025 · similarity 0.91)
- Provable Posterior Sampling with Denoising Oracles via Tilted Transport (NeurIPS 2024 · similarity 0.91)
- Fast samplers for Inverse Problems in Iterative Refinement models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.