Research map / Image generation / Fast diffusion sampling
Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LD-SMC, a sequential Monte Carlo sampler in the latent space of diffusion models that uses auxiliary observations along the reverse diffusion trajectory to solve inverse problems such as inpainting, deblurring and super-resolution.
Problem
Solving image inverse problems with latent diffusion models, where encoder-decoder nonlinearity and the sequential sampling process make conditioning on the corrupted observation hard, especially for inpainting.
Approach
Augments the latent diffusion model with per-timestep auxiliary observations y_t generated from the decoded latent, derives approximate target distributions and a novel Gaussian proposal combining the posterior-mean likelihood gradient and the auxiliary-observation gradient, and samples with SISR/SMC inside one blocked Gibbs step; works with nonlinear operators and LDMs.
Key results
- On ImageNet box inpainting, LD-SMC (5 particles) reaches FID 50.67 vs. 65.04 for LDPS and 64.74 for LTDS (lower is better).
- On ImageNet free-form inpainting, LD-SMC (5 particles) reaches FID 36.18 vs. 44.15 for ReSample, the best baseline.
- On FFHQ Gaussian deblurring, LD-SMC (1 particle) gets FID 29.19 vs. 29.30 for LDPS, roughly comparable; ReSample and LatentDAPS achieve the best PSNR/SSIM at the cost of perceptual quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LD-SMC instead of LTDS | CIN256-V2 latent diffusion · ImageNet test, box inpainting | FID: 64.74 -> 50.67 | Helped |
| Use LD-SMC instead of ReSample | CIN256-V2 latent diffusion · ImageNet test, free-form inpainting | FID: 44.15 -> 36.18 | Helped |
| Use LD-SMC instead of ReSample | CIN256-V2 latent diffusion · ImageNet test, Gaussian deblurring | FID: 46.45 -> 52.29 | Hurt |
| Use LD-SMC instead of LTDS | CIN256-V2 latent diffusion · ImageNet test, 8× super-resolution | FID: 59.12 -> 57.89 | Helped |
| Use LD-SMC instead of LTDS | VQ-4 latent diffusion · FFHQ test, box inpainting | FID: 39.57 -> 33.87 | Helped |
| Use LD-SMC instead of PSLD | VQ-4 latent diffusion · FFHQ test, free-form inpainting | FID: 36.26 -> 33.60 | Helped |
| Use LD-SMC instead of LDPS | VQ-4 latent diffusion · FFHQ test, Gaussian deblurring | FID: 29.30 -> 29.47 | Hurt |
| Use LD-SMC instead of LDPS | VQ-4 latent diffusion · FFHQ test, 8× super-resolution | FID: 29.64 -> 30.62 | Hurt |
| Increase particle count from 1 to 5 (ablation) | LD-SMC on ImageNet · ImageNet test, box inpainting | FID: 51.49 -> 50.67 | Helped |
| Increase particle count from 1 to 5 (ablation) | LD-SMC on ImageNet · ImageNet test, free-form inpainting | FID: 38.21 -> 36.18 | Helped |
| Increase particle count from 1 to 5 (ablation) | LD-SMC on FFHQ · FFHQ test, box inpainting | FID: 33.37 -> 33.87 | Hurt |
| Increase particle count from 1 to 5 (ablation) | LD-SMC on FFHQ · FFHQ test, free-form inpainting | FID: 33.67 -> 33.60 | Helped |
Surprising findings
- Tuning the proposal's auxiliary-observation weight s trades FID against PSNR, and setting s=0 (DPS-like proposal) significantly reduces perceptual quality.
- Increasing the number of particles sometimes improved FID only, not consistently across tasks, which the authors attribute to the small particle counts tested.
Limitations
- Sampling time and memory grow with the number of particles and Gibbs iterations.
- Asymptotic guarantees rely on assumptions (bounded gradients of decoder and diffusion model) that may not hold in practice.
- Hyperparameters (s, kappa, rho) are tuned per task via grid search using FID on reference images.
Experimental setup
Datasets: ImageNet FFHQ
Benchmarks: ImageNet FFHQ
Metrics: FID NIQE LPIPS PSNR SSIM
Baselines: LDPS LTDS ReSample PSLD PFLD LatentDAPS
Builds on: DDIM TDS (twisted diffusion sampler) FPS (filtering posterior sampling) DPS PSLD
Closest papers by meaning
- Solving Linear-Gaussian Bayesian Inverse Problems with Decoupled Diffusion Sequential Monte Carlo (ICML 2025 · similarity 0.90)
- Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion (CVPR 2024 · similarity 0.90)
- Self-diffusion for Solving Inverse Problems (NeurIPS 2025 · similarity 0.90)
- Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency (ICLR 2024 · similarity 0.90)
- Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models (ICML 2024 · similarity 0.89)
- Fast samplers for Inverse Problems in Iterative Refinement models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.