Research map / Image generation / Fast diffusion sampling
Flow Map Language Models: One-step Language Modeling via Continuous Denoising
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes flow language models over one-hot embeddings with cross-entropy denoiser training, and distills them into flow map language models (FMLM) that generate competitively in one step.
Problem
Discrete diffusion language models degrade sharply in the few-step regime because factorized approximations of the reverse transition ignore token correlations; continuous diffusion LMs have been thought to underperform.
Approach
Continuous linear flow over one-hot token embeddings with a softmax denoiser trained by cross-entropy; a two-time denoiser lives on the simplex, enabling cross-entropy/KL semigroup distillation of a flow map for one-step generation, plus a decoding-error-rate time reparameterization.
Key results
- FMLM one-step Gen. PPL 119.34 on LM1B (entropy 4.16), versus 1217-1429 for MDLM/Duo distilled baselines at one step and 292.94 for Duo+Di4C.
- FLM at 1024 steps reaches Gen. PPL 96.91 on LM1B and 62.23 on OWT, beating Duo (98.14, 77.69) and MDLM (109.21, 105.15) on LM1B and OWT.
- On Sudoku unconditional generation, FMLM reaches 9.38% validity in one step versus 0.00% for MDLM+SDTT and Duo+DCD.
Surprising findings
- Continuous one-hot flows with cross-entropy denoising match or beat state-of-the-art discrete diffusion, challenging the belief that continuous diffusion LMs underperform.
- Discrete diffusion guidance in logit space collapses at guidance scale 10 or higher, while the continuous autoguidance remains stable up to scale 100.
Limitations
- One-hot representation adds about 30% time and memory cost per training step over embedding diffusion methods.
- One-step generation quality still lags the 8-step quality of FLM and some few-step baselines; further distillation is needed.
- Evaluation relies on generative perplexity and entropy rather than validation perplexity, which requires auxiliary training for FLM.
Experimental setup
Datasets: LM1B OpenWebText Sudoku-Extreme-Full AG News CoLA IMDb TweetEval-Offensive
Benchmarks: LM1B OpenWebText Sudoku validity reward-guided generation benchmarks (AG News, CoLA, IMDb, TweetEval-Offensive)
Metrics: generative perplexity unigram entropy Self-BLEU validity reward score
Baselines: MDLM Duo RDLM CANDI MDLM+SDTT MDLM+Di4C Duo+DCD Duo+Di4C
Builds on: flow matching stochastic interpolants flow map matching MeanFlow progressive distillation
Closest papers by meaning
- LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling (NeurIPS 2026 · similarity 0.93)
- Continuous Diffusion Model for Language Modeling (NeurIPS 2025 · similarity 0.91)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.90)
- Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.90)
- ELF: Embedded Language Flows (NeurIPS 2026 · similarity 0.90)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.