Research map / Image generation / Fast diffusion sampling
Distillation of Discrete Diffusion through Dimensional Correlations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Di4C, a mixture-of-product-models denoiser for discrete diffusion plus distillation losses that capture dimensional correlations, enabling few-step sampling from pretrained teachers on CIFAR-10, ImageNet VQ tokens, and OpenWebText.
Problem
Discrete diffusion models use dimensionally independent (product) denoisers that need many sampling steps because they cannot capture dependencies between dimensions; few-step sampling therefore suffers large errors.
Approach
Model the denoiser as a mixture (expectation over a random latent lambda) of product distributions, which can represent dimensional correlations while keeping sampling cost nearly unchanged, and distill multi-step product teachers into it using distillation and consistency losses (Di4C) with theoretical TV bounds.
Key results
- On CIFAR-10 with 10 steps, the Di4C student improves FID from 32.61 (teacher) to 20.64.
- On ImageNet 256x256, 4-step Di4C models closely match 8-step teacher FID-IS, roughly a 2x speed-up.
- On OpenWebText, sdtt-6 + di4c achieves higher Self-BLEU diversity than sdtt-7 at equal steps with comparable generative perplexity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill the teacher into a Di4C student | U-Net teacher · CIFAR-10, 10 steps | FID: 32.61 -> 20.64 | Helped |
| Distill the teacher into a Di4C student | U-Net teacher · CIFAR-10, 10 steps | IS: 7.59 -> 8.29 | Helped |
| Distill the teacher into a Di4C student | U-Net teacher · CIFAR-10, 20 steps | FID: 12.36 -> 9.77 | Helped |
| Distill the teacher into a Di4C student | U-Net teacher · CIFAR-10, 20 steps | IS: 8.55 -> 8.52 | No clear effect |
| Use the student for the first half of denoising, then switch to the teacher | U-Net teacher · CIFAR-10, 20 steps | FID: 12.36 -> 9.47 | Helped |
| Distill the teacher into a Di4C student | U-Net teacher · CIFAR-10, 40 steps | FID: 8.01 -> 9.66 | Hurt |
| Distill the teacher into a Di4C student | U-Net teacher · CIFAR-10, NFE 20; 20{ +0+0} | FID: 14.42 -> 11.81 | Helped |
| Add predictor-corrector steps instead of allocating the NFE budget to denoising | CIFAR-10 teacher and student · CIFAR-10, NFE 20 | FID | Hurt |
| Reduce teacher sampling from 8 steps to 4 steps using Di4C | MaskGIT teacher · ImageNet 256×256 | FID-IS | No clear effect |
| Apply one round of Di4C distillation | sdtt-6 · OpenWebText, unconditional generation | generative perplexity | Helped |
| Apply Di4C distillation | sdtt-6 · OpenWebText, conditional generation | Self-BLEU | Helped |
| Use mixture modeling instead of a product model | Di4C · ImageNet, 4 steps | latency: 2.0741\spm{0.0035} -> 2.0734\spm{0.0043} | No clear effect |
Surprising findings
- Product-model analytical sampling has an Omega(1/N) TV error even in a two-dimensional example, so many steps are required.
- Empirical speed-ups are only about 2x despite theory suggesting one-step distillation is possible.
Limitations
- Empirical gains are around 2x fewer steps, far from the one-step theoretical promise.
- Theory is largely informal and relies on regularity assumptions and perfect marginal teachers.
- Loss functions and mixture design require tuning of lambda conditioning and alpha_t mixing.
Experimental setup
Datasets: CIFAR-10 ImageNet OpenWebText WebText
Benchmarks: CIFAR-10 ImageNet 256x256 OpenWebText
Metrics: FID Inception Score precision recall generative perplexity Self-BLEU latency
Baselines: teacher product model SDTT MaskGIT hybrid teacher-student
Builds on: SDTT MaskGIT MDLM analytical sampling
Closest papers by meaning
- DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents (ICML 2024 · similarity 0.89)
- Data-free Distillation of Diffusion Models with Bootstrapping (ICML 2024 · similarity 0.89)
- DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture (CVPR 2025 · similarity 0.88)
- Distilling ODE Solvers of Diffusion Models into Smaller Steps (CVPR 2024 · similarity 0.88)
- Multistep Distillation of Diffusion Models via Moment Matching (NeurIPS 2024 · similarity 0.88)
- Efficient Dataset Distillation via Minimax Diffusion (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.