Research map / Image generation / Fast diffusion sampling
ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ItDPDM, a Poisson-noise discrete diffusion model trained with an information-theoretic Poisson Reconstruction Loss that gives an exact relation to the data NLL, evaluated on synthetic discrete distributions, CIFAR-10 and Lakh MIDI.
Problem
Generative modeling of non-negative discrete data (e.g., symbolic music, images) where existing diffusion models rely on continuous embeddings or variational bounds that give inaccurate likelihoods.
Approach
Replace Gaussian noising with a Poisson noise channel and train the denoiser with a Poisson Reconstruction Loss (a Bregman divergence) that satisfies an I-MPRL identity linking the loss derivative to mutual information, yielding an exact (non-variational) NLL expression estimated via importance-sampling quadrature.
Key results
- On CIFAR-10 test NLL, ItDPDM (Poisson + PRL) reaches 0.18 with DDPM and 0.17 with IDDPM backbones, versus 0.44/0.48 for Gaussian + MSE.
- On synthetic discrete data, ItDPDM achieves lower Wasserstein-1 than DDPM on all six discrete distributions (e.g., PoissMix 0.99 vs 3.76) and beats LTJ on 4 of 6 datasets.
- On image generation, ItDPDM FID is 0.18 dB increase vs DDPM, second behind IDDPM; on music, D3PM obtains the lowest FAD while ItDPDM (0.64) outperforms LTJ (0.66).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ItDPDM instead of DDPM | Synthetic discrete datasets · PoissMix | WD: 3.76±0.32 -> 0.99±0.15 | Helped |
| Use ItDPDM instead of DDPM | Synthetic discrete datasets · PoissMix | NLL: 4.24 -> 3.72 | Helped |
| Use ItDPDM instead of DDPM | Synthetic discrete datasets · ZIP | WD: 2.31±0.66 -> 0.56±0.43 | Helped |
| Use ItDPDM instead of DDPM | Synthetic discrete datasets · ZIP | NLL: 1.67 -> 2.22 | Hurt |
| Use ItDPDM instead of DDPM | Synthetic discrete datasets · NBinomMix | WD: 4.89±0.59 -> 1.39±0.37 | Helped |
| Use ItDPDM instead of DDPM | Synthetic discrete datasets · BNB | NLL: 2.56 -> 1.87 | Helped |
| Use ItDPDM instead of DDPM | Synthetic discrete datasets · Zipf | WD: 1.51±0.53 -> 0.48±0.13 | Helped |
| Use ItDPDM instead of DDPM | Synthetic discrete datasets · Zipf | NLL: 1.34 -> 1.70 | Hurt |
| Use ItDPDM instead of DDPM | Synthetic discrete datasets · YS | WD: 0.32±0.12 -> 0.14±0.03 | Helped |
| Use Poisson noising with PRL instead of Gaussian noising with MSE | DDPM and IDDPM backbones · CIFAR-10 test subset | NLL: 0.44 -> 0.18; 0.48 -> 0.17 | Helped |
| Replace Gaussian MSE with Gaussian PRL (ablation) | DDPM and IDDPM backbones · CIFAR-10 test subset | NLL: 0.44 -> 0.27; 0.48 -> 0.32 | Helped |
| Replace Poisson PRL with Poisson MSE (ablation) | DDPM and IDDPM backbones · CIFAR-10 test subset | NLL: 0.23 -> 0.18; 0.22 -> 0.17 | Hurt |
Surprising findings
- Gaussian + PRL also outperforms Gaussian + MSE, suggesting the PRL objective helps even for Gaussian noising on discrete data.
- Raw metrics are lower than reported values in IDDPM and LTJ due to limited training (600 epochs) and reduced denoising steps (100), so absolute numbers are not comparable to prior work.
Limitations
- Acknowledged as a proof of concept; does not match state-of-the-art on real-world generation tasks due to limited training (600 vs 3000+ epochs) and limited tuning.
- Reduced denoising steps (100 instead of 1000) and fixed logistic sampling parameters limit performance.
- Evaluation limited to unconditional generation on the training distribution; no conditional or OOD evaluation.
Experimental setup
Datasets: CIFAR-10 Lakh MIDI PoissMix ZIP NBinomMix BNB Zipf Yule-Simon Gamma Log-Normal Lomax Half-Cauchy
Benchmarks: CIFAR-10 Lakh MIDI PoissMix ZIP NBinomMix BNB Zipf Yule-Simon
Metrics: negative log-likelihood Wasserstein-1 distance FID SSIM FAD Consistency MSID WD LPIPS SC
Baselines: DDPM IDDPM ITDiff LTJ D3PM Gaussian + MSE
Builds on: ITDiff (information-theoretic Gaussian diffusion) Learning-to-Jump (LTJ) DDPM IDDPM D3PM
Closest papers by meaning
- Towards Non-Asymptotic Convergence for Diffusion-Based Generative Models (ICLR 2024 · similarity 0.86)
- Bernoulli Flow Models: Self-Consistent Generative Modeling for Binary Data (NeurIPS 2026 · similarity 0.85)
- Information-Theoretic Discrete Diffusion (NeurIPS 2025 · similarity 0.85)
- Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms (NeurIPS 2025 · similarity 0.85)
- Information Theoretic Learning for Diffusion Models with Warm Start (NeurIPS 2025 · similarity 0.85)
- O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.