Research map / Image generation / Fast diffusion sampling

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Discrete diffusion solvers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LangFlow, a continuous embedding-space diffusion language model trained with Bregman-divergence flow matching and cross-entropy, with an ODE-based NLL bound, a Gumbel-based learned noise scheduler, and corrected self-conditioning, reaching PPL 30.0 on LM1B and 24.6 on OWT.

Problem

Continuous diffusion language models lag behind discrete diffusion in perplexity; embedding-space DLMs lack a principled training objective and an ODE-based likelihood bound, and their design choices (noise schedule, self-conditioning) are underexplored.

Approach

Connects cross-entropy training of embedding-space DLMs to Bregman-divergence flow matching, derives an ODE-based NLL upper bound for evaluation, sets the noise schedule by an information-uniform principle yielding a learnable Gumbel scheduler over logNSR, and shows self-conditioning improves both PPL and Gen. PPL for continuous diffusion.

Key results

• LangFlow reaches PPL 30.0 on LM1B (best among DLMs) and 24.6 on OWT, with Gen. PPL 92.2 on LM1B and 36.5 on OWT.
• Self-conditioning reduces LangFlow's LM1B Gen. PPL from 154.2 to 81.5 and PPL from 49.0 to 30.0, while for MDLM it improves Gen. PPL but worsens PPL.
• On zero-shot transfer, LangFlow beats the AR Transformer on 4 of 7 benchmarks and MDLM on 3 of 7.

Surprising findings

• Self-conditioning improves both PPL and Gen. PPL for continuous diffusion, unlike the trade-off seen in discrete diffusion.
• MSE-based embedding regression (as in Plaid) causes token embedding collapse, motivating cross-entropy training.

Limitations

• Sample entropy is lower than discrete baselines, attributed to frequent content-word repetition rather than local degeneration, which the authors leave to future work.
• Plaid baseline uses its own architecture since it performs worse under the DiT architecture, complicating comparison.
• Results are limited to 130M-parameter models trained for 1M steps; scaling behavior is untested.

Experimental setup

Datasets: LM1B OpenWebText

Benchmarks: LM1B OpenWebText PTB Wikitext Lambada AG News Pubmed Arxiv

Metrics: PPL Gen. PPL entropy zero-shot perplexity

Baselines: MDLM SEDD UDLM Duo Plaid Diffusion-LM FLM D3PM DiffusionBert Transformer AR

Builds on: Flow Matching Plaid MDLM Variational Flow Matching Duo

Closest papers by meaning

• Flow Map Language Models: One-step Language Modeling via Continuous Denoising (NeurIPS 2026 · similarity 0.93)
• ELF: Embedded Language Flows (NeurIPS 2026 · similarity 0.92)
• Continuous Latent Diffusion Language Model (NeurIPS 2026 · similarity 0.91)
• Continuous Diffusion Model for Language Modeling (NeurIPS 2025 · similarity 0.90)
• SemanticDLM+: Improving Diffusion LLMs through Bias-variance Trade-off in Transition Kernel Design (NeurIPS 2026 · similarity 0.90)
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.90)