Research map / Image generation / Fast diffusion sampling
Continuous Diffusion Model for Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RDLM, a continuous diffusion language model that maps categorical data onto the hypersphere via the Fisher-Rao geometry, generalizes masked and uniform discrete diffusion, and trains with a simulation-free cross-entropy objective using radial symmetry.
Problem
Continuous diffusion models for discrete data underperform discrete diffusion models, and the connection between discrete diffusion and continuous flows on the statistical manifold of categorical distributions is poorly understood.
Approach
Reparameterize categorical data as points on the positive orthant of a hypersphere (via sqrt of probabilities), show discrete diffusion transition distributions are flows on this manifold, and build a generative diffusion on the sphere that generalizes masked and uniform diffusion. Training uses a radial-symmetry-based Riemannian normal approximation of bridge marginals to avoid simulation, plus a
Key results
- On Text8, RDLM reaches BPC <= 1.32, beating the best prior diffusion baseline MD4 (1.37) and any-order AR models.
- On LM1B, RDLM reaches PPL <= 28.44 with 110M params, outperforming MDLM-style discrete diffusion except MDLM at 27.04 and far ahead of Diffusion-LM (118.62).
- On CIFAR-10 (order-agnostic pixel tokens), RDLM achieves BPD <= 2.73, the lowest among compared models including MD4 (2.78) and PixelCNN++ variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RDLM instead of the strongest reported diffusion baseline, MD4 | RDLM · Text8 test | BPC: ≤ 1.37 -> ≤ 1.32 | Helped |
| Use RDLM instead of MDLM | RDLM · LM1B test | PPL: ≤ 27.04 -> ≤ 28.44 | Mixed |
| Use RDLM instead of Diffusion-LM | RDLM · LM1B test | PPL: ≤ 118.62 -> ≤ 28.44 | Helped |
| Use RDLM instead of MD4 | RDLM · CIFAR-10 test | BPD: ≤ 2.78 -> ≤ 2.73 | Helped |
| Use RDLM instead of Fisher-Flow | RDLM · Promoter DNA sequence design | MSE: 0.029 -> 0.027 | Helped |
| Use cross-entropy loss instead of mean squared error loss (ablation) | RDLM training · Language modeling | NLL | Helped |
| Apply importance sampling over training time (ablation) | RDLM training · Language modeling | likelihood | Helped |
| Add dimension splitting for large vocabularies (ablation) | RDLM · LM1B | likelihood | Helped |
| Use the masked-and-uniform mixture path together with dimension splitting (ablation) | RDLM · Large-vocabulary language modeling | likelihood | Helped |
| Approximate bridge transitions with a Riemannian normal instead of simulating the process (ablation) | RDLM training · Transition distribution approximation | MMD | No clear effect |
| Use simulation-free Riemannian-normal sampling instead of simulation-based training (ablation) | RDLM training · Training | speedup: ×50 speedup | Helped |
Surprising findings
- Directly training on large-vocabulary discrete data (without dimension splitting) fails to capture the distribution, attributed to high input dimensionality and sharp transitions.
- Cross-entropy loss converges faster and yields better NLL than the derived MSE drift objective.
Limitations
- Authors note LM1B result is slightly behind MDLM (27.04), so not uniformly best.
- Simulation-free approximation via Riemannian normal is approximate; calibration constant needed for small d.
Experimental setup
Datasets: Text8 One Billion Words Dataset CIFAR-10 promoter DNA sequence dataset
Benchmarks: Text8 One Billion Words CIFAR-10 promoter DNA design
Metrics: bits per character perplexity bits per dimension MSE generative perplexity entropy MMD
Baselines: MDLM SEDD D3PM MD4 Plaid BFN Diffusion-LM Discrete Flow Fisher-Flow DirichletFM
Builds on: discrete diffusion (D3PM) MDLM Riemannian diffusion mixture Fisher-Rao metric flow matching on statistical manifold
Closest papers by meaning
- Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models (ICLR 2025 · similarity 0.91)
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.91)
- Flow Map Language Models: One-step Language Modeling via Continuous Denoising (NeurIPS 2026 · similarity 0.91)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.90)
- LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling (NeurIPS 2026 · similarity 0.90)
- Discrete Modeling via Boundary Conditional Diffusion Processes (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.