Research map / Image generation / Autoregressive and latent generation
Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise
Zhenghao Lin, Yeyun Gong, yelong shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, Weizhu Chen
This paper tried a common change: More diffusion sampling steps
Increase the number of inverse diffusion steps · XSum · generation quality Improved
What 457 accepted papers report for this change
- 277 improved
- 77 no clear effect
- 26 mixed
- 77 hurt
- Optimality of FSQ Tokens for Continuous Diffusion for Categorical Data with Application to Text-to-Speech ICML 2026 · increase denoising steps from 5 to 25 for FSQ-original · SEED-TTS test-en · WER: 2.39% -> 2.00%Improved
- DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing ICML 2026 · increase denoising steps from 10 to 100 · AIST++ · latency (ms/frame): 26.26 -> 261.91Hurt
- Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers ICML 2026 · more diffusion sampling steps (25 to 100) for our method · ImageNet 256x256, 32 tokens · gFID: 2.89 -> 2.54Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add large-scale CPD pre-training (ablation) | GENIE · XSum | ROUGE-1: 38.9 -> 42.9 | Improved |
| Add large-scale CPD pre-training (ablation) | GENIE · CNN/DailyMail | ROUGE-1: 43.8 -> 45.6 | Improved |
| Add large-scale CPD pre-training (ablation) | GENIE · Gigaword | ROUGE-1: 43.7 -> 45.7 | Improved |
| Add large-scale CPD pre-training (ablation) | GENIE · CommonGen | SPICE: 20.6 -> 23.4 | Improved |
| Increase pre-training from 100w to 500w (ablation) | GENIE · XSum | ROUGE-1: 39.4 -> 41.2 | Improved |
| Set CPD paragraph proportion to 30% rather than 15%, 20%, 25%, 35%, or 40% (ablation) | GENIE · XSum | ROUGE-1: $40.24_{\pm0.03}$; $40.14_{\pm0.10}$; $40.24_{\pm0.10}$; $40.37_{\pm0.04}$; $40.29_{\pm0.06}$; $40.15_{\pm0.08}$ | Mixed |
| Use uniform instead of loss-aware timestep sampling (ablation) | GENIE · XSum | ROUGE-1: $40.06_{\pm0.04}$ -> $40.24_{\pm0.03}$; $40.18_{\pm0.03}$ -> $40.37_{\pm0.04}$ | Improved |
| Use diffusion generation rather than BART diverse beam search | GENIE vs BART · XSum | SELF-BLEU: 75.6 -> 29.3 | Improved |
| Increase the number of inverse diffusion steps (ablation) | GENIE · XSum | generation quality | Improved |
Closest papers by meaning
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.90)
- Energy-Based Diffusion Language Models for Text Generation (ICLR 2025 · similarity 0.90)
- TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings (AAAI 2025 · similarity 0.90)
- How to Train Your Latent Diffusion Language Model Jointly With the Latent Space (NeurIPS 2026 · similarity 0.89)
- Denoising Time Matters: Diverse Generation in Diffusion Language Models (NeurIPS 2026 · similarity 0.89)
- Compressed and Smooth Latent Space for Text Diffusion Modeling (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
