Research map / Image generation / Fast diffusion sampling
Outsourced Diffusion Sampling: Efficient Posterior Inference in Latent Spaces of Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes outsourced diffusion sampling: trains diffusion samplers via trajectory balance to sample posteriors in the noise space of GANs, VAEs, and flow priors, then pushes samples through the generator for amortized conditional generation.
Problem
Sampling from posterior distributions p(x|y) ∝ p_θ(x) r(x,y) under generative priors (GANs, VAEs, flows) is intractable, and existing methods (MCMC, adjoint matching, variational fine-tuning) are slow or restricted to specific model families.
Approach
Use the noise-outsourcing representation x = f(z) so that the posterior is pulled back to the Gaussian noise space as p(z)r(f(z),y); train a diffusion sampler in that space with the off-policy trajectory balance objective using only black-box reward evaluations, then transform samples via f.
Key results
- On CIFAR-10 with an I-CFM prior, outsourced diffusion reaches expected classifier log-probability -3.35 and FID 34.28, versus -3.09 / 19.45 for adjoint matching and -2.80 / 46.69 for latent HMC.
- On StyleGAN3 FFHQ prompts, outsourced diffusion averages ImageReward 1.23 versus -0.62 for HMC and -1.52 for the prior.
- On SD3 RLHF, outsourced diffusion reaches average ImageReward 1.27 versus 0.84 for CFG and 0.79 for the prior, with CLIP diversity 0.16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use outsourced diffusion sampling instead of the prior sampler | SN-GAN · CIFAR-10 | FID: 97.14 -> 68.12 | Helped |
| Use outsourced diffusion sampling instead of the prior sampler | I-CFM · CIFAR-10 | classifier log probability: -5.88 -> -3.35 | Helped |
| Use outsourced diffusion sampling instead of the prior sampler | I-CFM · CIFAR-10 | FID: 84.79 -> 34.28 | Helped |
| Use outsourced diffusion sampling instead of Adjoint Matching | I-CFM · CIFAR-10 | FID: 19.45 -> 34.28 | Hurt |
| Use outsourced diffusion sampling instead of HMC | StyleGAN3 · FFHQ | ImageReward: -0.62 -> 1.23 | Helped |
| Use outsourced diffusion sampling instead of the prior sampler | StyleGAN3 · FFHQ | diversity: 0.36 -> 0.26 | Hurt |
| Use outsourced diffusion sampling instead of CFG | Stable Diffusion 3 · Text-to-image RLHF | ImageReward: 0.84 -> 1.27 | Helped |
| Use outsourced diffusion sampling instead of the prior sampler | Stable Diffusion 3 · Text-to-image RLHF | CLIP diversity: 0.19 -> 0.16 | Hurt |
| Use outsourced diffusion sampling instead of the prior sampler | FoldFlow 2 · Protein structure | log diversity reward: -1.325 ± 0.014 -> 0.422 ± 0.225 | Helped |
| Use outsourced diffusion sampling instead of the prior sampler | FoldFlow 2 · Protein structure | pairwise TM-Score: 0.4480 ± 0.0044 -> 0.4407 ± 0.0706 | No clear effect |
| Tune the classifier-free-guidance weight | Stable Diffusion 3 · Text-to-image RLHF | ImageReward | Helped |
| Increase the classifier-free-guidance weight too much | Stable Diffusion 3 · Text-to-image RLHF | ImageReward | Hurt |
Surprising findings
- Adjoint matching gives better FID than outsourced diffusion on CIFAR-10 with I-CFM despite lower reward.
- Latent HMC on CIFAR-10 SN-GAN reaches higher reward than outsourced diffusion (-3.26 vs -3.84) while outsourced diffusion has better FID.
Limitations
- Adjoint matching and HMC outperform it on some metrics (e.g., FID for I-CFM posteriors).
- Protein experiment reports high variance across seeds (reward 0.422 ± 0.225).
- The SD3 RLHF experiments use ImageReward, a weak reward model relative to SD3, so gains may reflect reward hacking.
Experimental setup
Datasets: CIFAR-10 FFHQ
Benchmarks: CIFAR-10 FFHQ Stable Diffusion prompts (RLHF) protein structure diversity
Metrics: expected log reward FID CLIP diversity pairwise TM-Score ELBO
Baselines: Hamiltonian Monte Carlo Adjoint Matching DPS RTB CFG DDPO DPOK Random-walk MCMC
Builds on: trajectory balance adjoint matching relative trajectory balance flow matching hamiltorch
Closest papers by meaning
- Amortizing intractable inference in diffusion models for vision, language, and control (NeurIPS 2024 · similarity 0.90)
- Improved off-policy training of diffusion samplers (NeurIPS 2024 · similarity 0.89)
- Learning to Sample From Diffusion Models via Inverse Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation (NeurIPS 2025 · similarity 0.89)
- CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling (ICLR 2024 · similarity 0.89)
- NoiseCtrl: A Sampling-Algorithm-Agnostic Conditional Generation Method for Diffusion Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.