Research map / Image generation / Fast diffusion sampling
Information Theoretic Learning for Diffusion Models with Warm Start
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a Fisher-information/KL relation for arbitrary isotropic noise and an upper bound on diffusion NLL with log-SNR parameterization, importance sampling, and warm-up dequantization noise, reporting state-of-the-art ImageNet bits/dim.
Problem
Likelihood estimation with perturbation-based diffusion models is slow to converge and theoretically under-understood; existing information-theoretic bounds are looser than ELBO, and dequantization/noise choices create train-test gaps in NLL.
Approach
Extends the de Bruijn / KL–Fisher relation from Gaussian to arbitrary isotropic noise, yielding a pointwise upper bound on NLL as a denoising score matching loss integrated over log-SNR with importance sampling, plus training-free warm-up noise for dequantization.
Key results
- Achieves 2.50 bits/dim on CIFAR-10 (VP with IS) and 2.49 (SP with IS) versus VDM 2.65.
- Reports state-of-the-art 3.01, 2.91, and 2.59 bits/dim on ImageNet-32, -64, and -128 respectively, beating W-PCDM at 3.32, 2.95, 2.64.
- Gaussian warm-up noise gives the best NLL on CIFAR-10 (2.49), with Uniform noise worst (2.52 for SP+IS); Uniform is worse on ImageNet-32 (3.09).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed variance-aware likelihood method with importance sampling | Diffusion models · ImageNet-32 | NLL (bits/dim): 3.32 -> 3.01 | Helped |
| Use the proposed variance-aware likelihood method with importance sampling | Diffusion models · ImageNet-64 | NLL (bits/dim): 2.95 -> 2.91 | Helped |
| Use the proposed variance-aware likelihood method with importance sampling | Diffusion models · ImageNet-128 | NLL (bits/dim): 2.64 -> 2.59 | Helped |
| Use the proposed variance-aware likelihood method with importance sampling | Diffusion models · CIFAR-10 | NLL (bits/dim): 2.35 -> 2.49 | Hurt |
| Use Laplace warm-up noise instead of Gaussian (ablation) | Ours (SP + IS) · CIFAR-10 | NLL (bits/dim): 2.49 -> 2.49 | No clear effect |
| Use logistic warm-up noise instead of Gaussian (ablation) | Ours (SP + IS) · CIFAR-10 | NLL (bits/dim): 2.49 -> 2.50 | Hurt |
| Use uniform warm-up noise instead of Gaussian (ablation) | Ours (SP + IS) · CIFAR-10 | NLL (bits/dim): 2.49 -> 2.52 | Hurt |
| Use Laplace warm-up noise instead of Gaussian (ablation) | Ours (SP + IS) · ImageNet-32 | NLL (bits/dim): 3.01 -> 3.00 | Helped |
| Use logistic warm-up noise instead of Gaussian (ablation) | Ours (SP + IS) · ImageNet-32 | NLL (bits/dim): 3.01 -> 3.02 | Hurt |
| Use uniform warm-up noise instead of Gaussian (ablation) | Ours (SP + IS) · ImageNet-32 | NLL (bits/dim): 3.01 -> 3.09 | Hurt |
| Use Gaussian warm-up noise instead of uniform (ablation) | Ours (SP + IS) · CIFAR-10 and ImageNet-32 | NLL (bits/dim): 2.52 -> 2.49; 3.09 -> 3.01 | Helped |
| Use hand-crafted importance sampling rather than a learned proposal (ablation) | Ours · CIFAR-10 | NLL | No clear effect |
Surprising findings
- A VE schedule attained only 3.27 bits/dim on CIFAR-10 in preliminary runs, so VE models were omitted.
- Larger warm-up noise improves NLL but slightly degrades FID.
Limitations
- Method only evaluates likelihood and cannot be used directly for sampling or generation.
- Dequantization bound, diffusion coefficients and variance configurations are not fully explored.
- Hyperparameter and architecture tuning was not explored due to resource limits.
Experimental setup
Datasets: CIFAR-10 ImageNet-32 ImageNet-64 ImageNet-128
Benchmarks: CIFAR-10 ImageNet-32 ImageNet-64 ImageNet-128
Metrics: NLL (bits/dim) FID
Baselines: VDM ScoreODE ScoreFlow Flow Matching Stochastic Interpolants i-DODE DiffEnc MuLAN BSI W-PCDM LP-PCDM
Builds on: VDM Variational diffusion models i-DODE W-PCDM de Bruijn identity
Closest papers by meaning
- Diffusion Models With Learned Adaptive Noise (NeurIPS 2024 · similarity 0.91)
- Conservation Laws for Diffusion Models (NeurIPS 2026 · similarity 0.89)
- Warm Diffusion: Recipe for Blur-Noise Mixture Diffusion Models (ICLR 2025 · similarity 0.89)
- Neural Diffusion Models (ICML 2024 · similarity 0.88)
- Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data (ICML 2024 · similarity 0.88)
- Diffusion Priors for Variational Likelihood Estimation and Image Denoising (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.