Research map / Image generation / Fast diffusion sampling
Glauber Generative Model: Discrete Diffusion Models via Binary Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Glauber Generative Model, a discrete diffusion model that reverses a one-token-at-a-time noising process by training a transformer to solve per-token binary classification problems, applied to language and image (VQGAN-token) generation.
Problem
Discrete diffusion models for tokens (language and image tokens) are typically trained with variational losses or regression on importance ratios, scaling poorly with vocabulary size and lagging autoregressive models.
Approach
Noise one token position at a time with a time-dependent forward process, so the reverse step is an exact reduction to binary classification (signal vs. noise) per vocabulary token, with cost linear in vocabulary size, and denoise via heat-bath (Glauber) dynamics.
Key results
- On OpenWebText, GGM (387M params) reaches generative perplexity 19.5 under GPT2-large evaluation, outperforming SEDD-medium (27.3) and MDLM (44.2) and nearly matching Plaid (1.3B, 19.7).
- On FFHQ 256x256 unconditional generation, GGM with MUSE tokens achieves FID 12.5, beating the StyleSwin baseline (15.0) and VQ-DDM variants.
- On CelebA-HQ, GGM achieves FID 9.8, outperforming several autoencoder baselines and VQ-DDM (w/ ReFiT) at 13.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use time-dependent Glauber dynamics with binary-classification training instead of existing discrete diffusion methods | GGM language generation · OpenWebText, unconditional generation, GPT-neo evaluator | generative perplexity: 25.2 -> 18.0 (SEDD-medium); 40.9 -> 18.0 (MDLM) | Helped |
| Use GGM instead of Plaid | language generation · OpenWebText, unconditional generation, GPT2-large evaluator | generative perplexity: 19.7 -> 19.5 | Helped |
| Use GGM instead of Plaid | language generation · OpenWebText, unconditional generation, GPT2-xl evaluator | generative perplexity: 19.7 -> 19.9 | Hurt |
| Use GGM instead of comparable-size autoregressive GPT2 models | language generation · OpenWebText, unconditional generation | generative perplexity: 12.4 -> 19.5 (GPT2-large evaluator); 13.0 -> 19.9 (GPT2-xl evaluator); 14.5 -> 18.0 (GPT-neo evaluator) | Hurt |
| Increase BERT-large Gibbs sampling from T=2048 to T=65536 | BERT-large with time-independent Gibbs sampling · OpenWebText, unconditional generation, GPT-neo evaluator | generative perplexity: 488.7 -> 26.4 | Helped |
| Use BERT-large time-independent Gibbs sampling instead of time-dependent GGM | language generation · OpenWebText, unconditional generation, GPT-neo evaluator | generative perplexity: 18.0 (GGM) vs 26.4 (BERT-large, T=65536) | Hurt |
| Use GGM instead of VQ-DDM with ReFiT | image generation · CelebA-HQ, 256×256 unconditional generation | FID: 13.2 -> 9.8 | Helped |
| Use GGM instead of VQGAN + GPT2 | image generation · CelebA-HQ, 256×256 unconditional generation | FID: 10.2 -> 9.8 | Helped |
| Use GGM instead of LDM-4 | image generation · CelebA-HQ, 256×256 unconditional generation | FID: 5.1 -> 9.8 | Hurt |
| Use GGM instead of BigGAN | image generation · FFHQ, 256×256 unconditional generation | FID: 12.4 -> 12.5 | Hurt |
| Use GGM instead of VQGAN + GPT2 | image generation · FFHQ, 256×256 unconditional generation | FID: 9.6 -> 12.5 | Hurt |
| Use GGM instead of LDM-4 | image generation · FFHQ, 256×256 unconditional generation | FID: 5.0 -> 12.5 | Hurt |
Surprising findings
- Time-independent Glauber dynamics with BERT-large Gibbs sampling yields generative perplexity around 487 at 2048 steps, far worse than GGM, even though GGM is compared at fewer effective visits.
- The authors note that default SEDD and MDLM samplers, when run in FP64 rather than FP32, give generative perplexity near 100, more than 3x the originally reported values.
Limitations
- Image FID lags state-of-the-art GANs, continuous diffusion and autoregressive models, attributed to using a general-purpose tokenizer with 1024 tokens.
- Language generation still lags GPT2 autoregressive models of comparable size.
- Noise schedule and Pi(phi) were not explored; only Pi(phi)=0.5 was used.
Experimental setup
Datasets: OpenWebText CelebA-HQ FFHQ
Benchmarks: OpenWebText generative perplexity CelebA-HQ FID FFHQ FID
Metrics: generative perplexity FID
Baselines: GPT2-medium GPT2-large GPT2-xl BERT-large Gibbs sampling Plaid SEDD-medium MDLM VQ-DDM VQ-Diffusion LDM-4 StyleSwin VQGAN + GPT2
Builds on: D3PM MDLM SEDD DiT MUSE
Closest papers by meaning
- Simplified and Generalized Masked Diffusion for Discrete Data (NeurIPS 2024 · similarity 0.89)
- Discrete Modeling via Boundary Conditional Diffusion Processes (NeurIPS 2024 · similarity 0.89)
- Scaling Inference Time Compute for Diffusion Models (CVPR 2025 · similarity 0.88)
- Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling (ICLR 2025 · similarity 0.88)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.88)
- A solvable model of learning generative diffusion: theory and insights (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.