Research map / Image generation / Fast diffusion sampling
Understanding the Generalization of Pretrained Diffusion Models on Out-of-Distribution Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that diffusion model latent spaces (DAE, D2C, LDM) invert out-of-distribution images better than GAN latent spaces, gives a theoretical argument based on absent prior holes and a hypersphere geometry, and proposes geodesic-based manipulation and few-shot generation.
Problem
GANs such as StyleGAN struggle to invert and edit out-of-distribution (OOD) images because their latent spaces have prior holes and poor geometry, limiting downstream use on non-face or non-FFHQ data.
Approach
Diffusion latent spaces (via forward-process inversion) match the N(0,I) prior by construction, so they avoid prior holes; their Gaussian geometry projects onto a bounded hypersphere, enabling geodesic traversal for manipulation and geodesic-based few-shot sampling from a handful of inverted images.
Key results
- Diffusion-based OOD inversion (DAE 32.14 PSNR on ImageNet) substantially outperforms the best GAN inversion baseline SPHInX (27.63 PSNR on ImageNet, 21.79 on ImageNet Sketches).
- Manipulation direction parallelism (average angle) is about 0.07-0.12 radians for diffusion models versus 0.89-1.12 radians for SPHInX in StyleGAN space.
- Few-shot FS-DAE obtains FID 63.12 on Babies and 35.51 on Sunglasses, roughly matching Latent Learner (63.31, 35.64) and beating Ojha et al. (74.39, 42.13) without per-set fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use diffusion-space DAE inversion instead of SPHInX GAN inversion | OOD image inversion · ImageNet | PSNR: 27.63 -> 32.14 | Helped |
| Use diffusion-space DAE inversion instead of SPHInX GAN inversion | OOD image inversion · ImageNet Sketches | PSNR: 21.79 -> 31.77 | Helped |
| Use diffusion-space DAE inversion instead of SPHInX GAN inversion | OOD image inversion · DSPRITES | PSNR: 27.69 -> 32.09 | Helped |
| Use diffusion-space DAE inversion instead of SPHInX GAN inversion | OOD image inversion · QuickDraw | PSNR: 23.06 -> 32.04 | Helped |
| Use diffusion-space DAE traversal instead of SPHInX-BRGM | Image colorization · LHQ-256 | PSNR: 27.34 -> 32.12 | Helped |
| Use diffusion-space DAE traversal instead of SPHInX-BRGM | Image super-resolution · LHQ-256 | PSNR: 27.13 -> 32.08 | Helped |
| Use diffusion-space D2C traversal instead of SPHInX-BRGM | Image inpainting · LHQ-256 | PSNR: 26.16 -> 32.12 | Helped |
| Use FS-DAE instead of Latent Learner | Few-shot generation · Babies | FID: 63.31 -> 63.12 | Helped |
| Use FS-DAE instead of Latent Learner | Few-shot generation · Sunglasses | FID: 35.64 -> 35.51 | Helped |
| Use FS-DAE instead of Latent Learner | Few-shot generation · Sketches | FID: 35.59 -> 36.21 | Hurt |
| Use FS-DAE instead of Latent Learner | Few-shot generation · Bitmoji | FID: 64.50 -> 64.43 | Helped |
| Use DAE instead of SPHInX-BRGM | Non-linear image traversal · LHQ-256 colorization | SSIM: 0.8823 -> 0.968 | Helped |
Surprising findings
- The few-shot method is a naive geometric approach (sampling on geodesics between inverted latents) yet matches state-of-the-art fine-tuned methods in FID on several sets, and it is slightly worse than Latent Learner on Sketches and Bitmoji.
- The claimed prior-hole advantage of diffusion models is argued theoretically, while the paper's own lemma shows divergence objectives cannot fix prior holes for Gaussian priors.
Limitations
- Few-shot generation uses only the linear (geodesic) version as a preliminary study.
- Theoretical lemmas are argued heuristically (e.g., Lemma 1 and 2.1 rely on approximations and concentration inequalities) rather than fully proven.
- Stable Diffusion is not analyzed directly; results on LDM (FFHQ) are extrapolated to it.
Experimental setup
Datasets: FFHQ CelebA ImageNet ImageNet Sketches DSPRITES QuickDraw LHQ-256 MS-COCO Babies Sunglasses Sketches Bitmoji
Benchmarks: ImageNet ImageNet Sketches DSPRITES QuickDraw LHQ-256 Babies Sunglasses Sketches Bitmoji
Metrics: PSNR SSIM FID Proxy A-Distance average angle between manipulation directions
Baselines: SPHInX StyleGAN-XL BigGAN PULSE BRGM Latent Learner Ojha et al. 2021
Builds on: Diffusion Autoencoders D2C Latent Diffusion Models SPHInX DiffusionCLIP
Closest papers by meaning
- Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation (ICML 2025 · similarity 0.90)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.89)
- Outsourced Diffusion Sampling: Efficient Posterior Inference in Latent Spaces of Generative Models (ICML 2025 · similarity 0.88)
- Neural Diffusion Models (ICML 2024 · similarity 0.88)
- The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noise (ICLR 2025 · similarity 0.88)
- Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.