Research map / Image generation / Fast diffusion sampling
The Emergence of Reproducibility and Consistency in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that diffusion models trained on the same data reproduce the same noise-to-image mapping under deterministic samplers, with this reproducibility arising from memorization (empirical score) or generalization (true distribution) regimes depending on training data size.
Problem
Why different diffusion models given the same initial noise and a deterministic sampler produce remarkably similar outputs, and what distribution they learn in different training data size regimes.
Approach
Introduces "consistent model reproducibility" with RP and GL scores, and shows it arises in two regimes: memorization (models converge to the empirical multi-delta score) and generalization (models converge to the underlying data score), with extensions to conditional models, inverse problems and fine-tuning.
Key results
- Across seven unconditional diffusion models trained on CIFAR-10 (differing in architecture, training, sampler, kernel), pairwise RP scores are high, with even the most dissimilar pairs at about 0.7 RP and 0.68 MAE.
- Two pretrained-model case: a model retrained on samples from a pretrained diffusion model achieves RP score 0.80 with it, supporting convergence to the same underlying distribution.
- Stable Diffusion checkpoints v1-3 and v1-4 (sharing training subset) show reproducibility score 0.63, while different initial noise with the same prompt yields scores near zero.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a second diffusion model on samples generated by a pretrained model | EDM on CIFAR-10 · CIFAR-10 | RP score: RP Score=0.80 | Helped |
| Compare different unconditional diffusion models using the same initial noise | Unconditional diffusion models · CIFAR-10 | RP score: 0.7 | Helped |
| Compare different unconditional diffusion models using the same initial noise | Unconditional diffusion models · CIFAR-10 | MAE score: 0.68 | Helped |
| Train on the same LAION-5B subset (v1-3 and v1-4 share the subset) | Stable Diffusion · Prompt: “a photograph of an astronaut riding a horse” | Reproducibility score: 0.63 | Helped |
| Use sequentially trained Stable Diffusion versions with overlapping datasets | Stable Diffusion v1-1, v1-2, and v1-3 · Prompt: “a photograph of an astronaut riding a horse” | Reproducibility score: below 0.21 | Helped |
| Change the initial noise while keeping the prompt fixed | Stable Diffusion · Prompt: “a photograph of an astronaut riding a horse” | Reproducibility score | Hurt |
| Compare models with different network architecture types | Diffusion models solving inverse problems with DPS · CIFAR-10 image inpainting | RP score | Hurt |
| Make the inpainting task more challenging | Diffusion models solving inverse problems with DPS · CIFAR-10 image inpainting | Reproducibility score | Hurt |
Surprising findings
- Reproducibility is rare in GANs and VAEs, suggesting it is a distinctive property of diffusion models.
- Under DPS-based inverse problem solving, reproducibility holds within architecture families (U-Net vs transformer) but drops notably across them.
Limitations
- Analysis of memorization regime is restricted to small datasets; practical interest limited there.
- Reproducibility across architectures is weaker for inverse problems and not fully explained; left for future study.
- Theoretical analysis of generalization regime is empirical only; true natural image distribution not analyzed.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet MNIST LAION-5B
Benchmarks: CIFAR-10 ImageNet
Metrics: RP score GL score MAE score score matching loss L_score
Baselines: GAN (wGAN, SNGAN) VAE (VAMP)
Builds on: DDPM score-based SDE EDM DPS Progressive distillation
Closest papers by meaning
- Understanding diffusion models requires rethinking (again) generalization (NeurIPS 2026 · similarity 0.93)
- On the Edge of Memorization in Diffusion Models (NeurIPS 2025 · similarity 0.92)
- Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data (ICML 2024 · similarity 0.92)
- Training Data Attribution in Diffusion Models via Mirrored Unlearning and Noise-Consistent Skew (NeurIPS 2026 · similarity 0.91)
- Elucidating the Exposure Bias in Diffusion Models (ICLR 2024 · similarity 0.90)
- Generalization through variance: how noise shapes inductive biases in diffusion models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.