Research map / Image generation / Autoregressive and latent generation
TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TEncDM, a continuous text diffusion model trained in the latent space of pre-trained BERT-style encoder outputs (encodings) with a transformer decoder, and analyzes encoder, decoder, noise schedule and self-conditioning effects.
Problem
Continuous diffusion for text is hard because token-embedding latents lack context, decoding from noisy latents is error-prone, and noise schedules and self-conditioning are poorly understood.
Approach
Train the diffusion model on contextual encodings from a frozen pre-trained language model instead of raw token embeddings, decode with a context-aware transformer decoder trained on corrupted latents, and use a tan-d noise schedule that adds more noise across all timesteps.
Key results
- On ROCStories, the BERT-encoding latent space achieves MAUVE 0.762 versus 0.600 for BERT embeddings, and perplexity 29.1 versus 48.9.
- TEncDM outperforms DiffuSeq, SeqDiffuSeq, GENIE and AR-Diffusion on QQP, XSum and Wiki-Auto; e.g., XSum ROUGE-L 26.8-27.1 range vs AR-Diffusion 20.8 for the T5 and RoBERTa variants.
- Scaling the diffusion model from 105M to 223M parameters improves Wikipedia MAUVE from 0.874 to 0.909.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BERT encodings instead of BERT token embeddings as diffusion latents (ablation) | TEncDM · ROCStories | MAUVE: .600_{.016} -> .762_{.043} | Helped |
| Use BERT encodings instead of BERT token embeddings as diffusion latents (ablation) | TEncDM · Wikipedia | MAUVE: .378_{.055} -> .874_{.011} | Helped |
| Train the decoder on corrupted latents (ablation) | TEncDM with Transformer decoder · Wikipedia | MAUVE: .526_{.025} -> .874_{.011} | Helped |
| Use a contextual Transformer decoder instead of an MLP decoder (ablation) | TEncDM with corrupted decoder inputs · ROCStories | perplexity: 31.2_{.33} -> 29.1_{.89} | Helped |
| Replace tan-9 with cosine noise scheduling (ablation) | TEncDM · ROCStories | MAUVE: .762_{.043} -> .098_{.011} | Hurt |
| Replace tan-9 with sqrt noise scheduling (ablation) | TEncDM · Wikipedia | MAUVE: .874_{.011} -> .325_{.037} | Hurt |
| Scale the diffusion model from base (105M) to large (223M) (ablation) | TEncDM · Wikipedia | perplexity: 104.4_{2.1} -> 84.5_{0.8} | Helped |
| Use TEncDM instead of GPT2-small FT | Unconditional generation · ROCStories | MAUVE: .739_{.031} -> .762_{.043} | Helped |
| Use TEncDM instead of GPT2-small FT | Unconditional generation · Wikipedia | MAUVE: .708_{.025} -> .874_{.011} | Helped |
| Use self-conditioning (ablation) | TEncDM · ROCStories | MAUVE | Mixed |
| Use self-conditioning and increase generation steps beyond 50 (ablation) | TEncDM · ROCStories | MAUVE | Hurt |
| Use T5-base as the encoder (ablation) | TEncDM · QQP | ROUGE-L: 56.4 -> 57.3 | Helped |
Surprising findings
- Self-conditioning increases prediction magnitude with more sampling steps, causing quality to drop beyond about 50 steps for the ROCStories model.
- The fine-tuned GPT-2-small baseline obtains better perplexity than TEncDM, though the perplexity comparison uses GPT-2-large as judge.
Limitations
- Encoder is frozen and not trained jointly with the decoder and diffusion model.
- Latent dimensionality scales with sequence length, so training slows for long texts.
- Best diffusion encoder differs by task, requiring per-task selection.
Experimental setup
Datasets: ROCStories Wikipedia (ROOTS subset) QQP XSum Wiki-Auto
Benchmarks: ROCStories Wikipedia QQP XSum Wiki-Auto
Metrics: perplexity memorization diversity MAUVE ROUGE-L ROUGE-1/2/L BERTScore BLEU-4
Baselines: DiffuSeq SeqDiffuSeq GENIE AR-Diffusion Transformer-base FLAN-T5-base GPT2-small FT
Builds on: DiffuSeq self-conditioning (Analog Bits / CDCD) Gaussian diffusion models classifier-free guidance
Closest papers by meaning
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.90)
- Theoretical Benefit and Limitation of Diffusion Language Model (NeurIPS 2025 · similarity 0.90)
- Energy-Based Diffusion Language Models for Text Generation (ICLR 2025 · similarity 0.90)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.90)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.90)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.