Research map / Image generation / Fast diffusion sampling
Informed Correctors for Discrete Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an informed predictor-corrector sampler for masked discrete diffusion that uses a hollow transformer to resample low-confidence tokens, plus a combined ELBO training objective (HollowDiff), evaluated on a synthetic Markov chain, text8, and tokenized ImageNet 256.
Problem
Sampling from masked discrete diffusion models with few steps accumulates approximation errors, and the existing forward-backward and uninformed correctors fix errors only by random re-masking, which is inefficient.
Approach
An adaptive random-scan Gibbs corrector that selects low-confidence non-mask tokens via Gumbel-top-k and resamples them in parallel from the conditional denoiser, made single-pass via a hollow transformer; the hollow model is trained with a combined objective averaging the masked and non-masked ELBO terms.
Key results
- On ImageNet 256x256 with 17 steps, MD4 + informed corrector (hybrid, 400M params) reaches FID 5.78 versus 6.28 for MD4 alone; HollowDiff + informed corrector reaches FID 6.26 at 230M params.
- The uninformed forward-backward corrector increases MD4's FID from 6.28 to 7.41 on ImageNet, while the informed corrector improves it.
- On the synthetic Markov chain, ancestral sampling with the informed corrector yields the lowest error rate across all tested NFE settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an informed corrector to HollowDiff (ablation) | HollowDiff · ImageNet 256×256 | FID: 10.24 -> 6.26 | Helped |
| Add an uninformed corrector (ablation) | HollowDiff · ImageNet 256×256 | FID: 10.24 -> 10.93 | Hurt |
| Add an informed corrector (ablation) | MD4 · ImageNet 256×256 | FID: 6.28 -> 5.78 | Helped |
| Replace MD4 predictor with HollowDiff predictor while using an informed corrector | MD4 + informed corrector · ImageNet 256×256 | FID: 5.78 -> 6.26 | Hurt |
| Add an uninformed corrector | MD4 · ImageNet 256×256 | FID: 6.28 -> 7.41 | Hurt |
| Use margin-based confidence instead of log-likelihood confidence (ablation) | HollowDiff + informed corrector · ImageNet 256×256 | FID: 6.45 -> 6.26 | Helped |
| Use margin-based confidence instead of log-likelihood confidence (ablation) | MD4 + informed corrector · ImageNet 256×256 | FID: 5.85 -> 5.78 | Helped |
| Increase the number of parallel updates k (ablation) | MD4 + informed corrector · ImageNet 256×256 | FID | Mixed |
| Use too many parallel updates (k ≥ 32) (ablation) | Informed corrector · ImageNet 256×256 | FID | Hurt |
| Use a final argmax update (ablation) | Single-network informed corrector · ImageNet 256×256 | FID | Helped |
| Use HollowDiff instead of a standard transformer for the predictor, without correctors (ablation) | ImageNet diffusion model · ImageNet 256×256 | FID: 6.28 -> 10.24 | Hurt |
| Combine the MD4 predictor with a separate HollowDiff informed corrector | MD4 · ImageNet 256×256 | FID: 6.28 -> 5.78 | Helped |
Surprising findings
- The uninformed corrector performs worse than using no corrector at all in the Markov chain experiment, and also worsens MD4's FID on ImageNet (6.28 to 7.41).
- Switching the denoiser to a hollow transformer degrades predictor-only quality (FID 10.24 vs 6.28 for MD4), yet the informed corrector recovers it.
Limitations
- Informed correctors for absorbing diffusion require the hollow transformer architecture, so they cannot be applied directly to arbitrary pretrained models.
- Jointly learning predictor and corrector trades off denoiser and corrector quality; separate models perform better in practice.
- Too large a number of parallel corrector updates k causes update collisions and worse FID.
Experimental setup
Datasets: text8 ImageNet 256x256 (tokenized with VQ tokenizer) synthetic Markov chain (hidden Markov model)
Benchmarks: text8 word error rate ImageNet 256x256 FID synthetic Markov chain error rate
Metrics: FID IS precision recall word error rate sequence error rate
Baselines: MD4 ancestral sampling MD4 + uninformed corrector ReMDM sampler MaskGIT DPC VQ-Diffusion
Builds on: MD4 forward-backward corrector (Campbell et al.) hollow transformer predictor-corrector sampling MaskGIT
Closest papers by meaning
- Encoder-Decoder Diffusion Language Models for Efficient Training and Inference (NeurIPS 2025 · similarity 0.90)
- Learnable Sampler Distillation for Discrete Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
- Learn from Your Mistakes: Self-Correcting Masked Diffusion Models (NeurIPS 2026 · similarity 0.88)
- Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms (NeurIPS 2025 · similarity 0.88)
- Evaluating the design space of diffusion-based generative models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.