Research map / Image generation / Fast diffusion sampling
A Mixture-Based Framework for Guiding Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MGDM, a Gibbs-sampling-based method that samples from a mixture of intermediate posterior approximations for training-free diffusion-prior inverse problem solving, evaluated on image restoration and audio source separation.
Problem
Training-free posterior sampling for Bayesian inverse problems using pre-trained denoising diffusion priors, where intermediate guidance likelihood terms are intractable and existing approximations are crude and unstable, especially for latent diffusion models.
Approach
Approximate each intermediate posterior by a mixture over earlier-timestep likelihood approximations (a mixture over time indices s), and sample each mixture component with a Gibbs sampler on a data-augmented extended distribution using noising/denoising steps plus a Gaussian variational update, so performance scales with the number of Gibbs steps.
Key results
- On FFHQ pixel-space with 100 steps and one Gibbs step, the method ranks first on most of the 10 image tasks by LPIPS (e.g., SR x4 0.09 and Gaussian deblur 0.12).
- On latent FFHQ, the method achieves the best LPIPS on 6 of 10 tasks, e.g., SR x4 0.14 vs 0.22 for ReSample.
- On Slakh source separation, the method reaches mean SI-SDRi 17.35 dB, above MSDM (16.48) and other training-free baselines, but below Demucs (17.73) with R=6 Gibbs steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MGDM instead of DPS | FFHQ pixel-space diffusion · FFHQ Gaussian Deblur | LPIPS: 0.17 -> 0.12 | Helped |
| Use MGDM instead of DPS | ImageNet pixel-space diffusion · ImageNet Motion Deblur | LPIPS: 0.40 -> 0.22 | Helped |
| Use MGDM instead of DAPS | FFHQ pixel-space diffusion · FFHQ HDR | LPIPS: 0.10 -> 0.12 | Hurt |
| Use MGDM instead of DPS | ImageNet pixel-space diffusion · ImageNet SR ×16 | LPIPS: 0.44 -> 0.55 | Hurt |
| Increase Gibbs steps R (ablation) | MGDM · FFHQ Phase retrieval | LPIPS | Helped |
| Increase Gibbs steps R (ablation) | MGDM · Slakh test source separation | SI-SDR_I | Helped |
| Use R=1 with more gradient steps rather than R=6 Gibbs steps at equivalent runtime (ablation) | MGDM · Slakh test source separation | SI-SDR_I | Helped |
| Sample the index near 0 consistently instead of sampling uniformly (ablation) | MGDM · Image inverse problems | reconstruction quality | Hurt |
| Compute the squared-norm expectation exactly instead of using a Monte Carlo estimator (ablation) | MGDM variational approximation · Image reconstruction | reconstruction quality | Hurt |
| Marginalize out x0 in the data augmentation (ablation) | MGDM alternative sampler · Image reconstruction | reconstruction quality | Hurt |
| Use MGDM instead of Resample | FFHQ latent diffusion · FFHQ Gaussian Deblur | LPIPS: 0.16 -> 0.18 | Hurt |
| Use MGDM instead of Demucs | Audio diffusion source separation · Slakh test, all stems | SI-SDR_I: 17.73 -> 17.35 | Hurt |
Surprising findings
- Sampling the mixture index deterministically near s=0 (the intuitively most accurate likelihood) degraded performance; uniform sampling over indices with a lower bound tau=10 worked better.
Limitations
- Gap to pixel-space performance on latent diffusion remains; weight sequence selection is important and not fully resolved.
- Does not extend to ODE-based samplers or DDIM.
- Higher memory cost due to vector-Jacobian product in the likelihood approximation.
Experimental setup
Datasets: FFHQ ImageNet Slakh2100
Benchmarks: FFHQ ImageNet Slakh test set
Metrics: LPIPS PSNR SSIM SI-SDRi
Baselines: DPS PGDM DDNM DiffPIR ReSample DAPS PnP-DM PSLD MSDM ISDM Demucs
Builds on: DPS DAPS PnP-DM MGPS RePaint
Closest papers by meaning
- Divide-and-Conquer Posterior Sampling for Denoising Diffusion priors (NeurIPS 2024 · similarity 0.94)
- A Variational Perspective on Solving Inverse Problems with Diffusion Models (ICLR 2024 · similarity 0.93)
- Learning Diffusion Priors from Observations by Expectation Maximization (NeurIPS 2024 · similarity 0.92)
- Diffusion Models With Learned Adaptive Noise (NeurIPS 2024 · similarity 0.92)
- Provable Posterior Sampling with Denoising Oracles via Tilted Transport (NeurIPS 2024 · similarity 0.91)
- End-to-end Learning of Gaussian Mixture Priors for Diffusion Sampler (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.