Research map / Probabilistic and scientific modeling / Variational and generative sampling
Discrete Flow Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Discrete Flow Matching, a CTMC-based flow framework for discrete data with general probability paths, closed-form probability velocities, and corrector sampling, scaled to 1.7B-parameter language and code models.
Problem
Flow matching and diffusion have excelled on continuous data but are limited for high-dimensional discrete data such as language, where autoregressive models dominate.
Approach
Generalizes discrete flows to a Flow Matching framework over arbitrary source-target couplings and convex schedulers, yielding a generating probability velocity in closed form via learned posteriors that has the same form as continuous flow matching; adds corrector sampling with provably valid stationary-distribution iterations.
Key results
- 1.7B DFM reaches 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1 and 20.6% Pass@10 on 1-shot MBPP.
- Conditional text generative perplexity of 9.7 (Llama-3 8B judge) vs 22.3 for a 1.7B autoregressive model and 8.3 for Llama-2 7B.
- On CIFAR10, achieves 3.63 FID at 1024 NFE, outperforming MaskGIT and the linear-scheduler discrete flow baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use conditional coupling instead of unconditional coupling (ablation) | Discrete Flow Matching · Conditional text generation | Llama-3 generative perplexity: 10.3 -> 9.7 | Helped |
| condition on the oracle solution length (ablation) | Discrete Flow Matching · HumanEval | Pass@1: 6.7 vs. 11.6 | Helped |
| use a cubic path scheduler with a=0, b=2 (ablation) | Discrete Flow Matching 150M text models · OpenWebText unconditional generation | GPT-2 generative perplexity | Helped |
| use smaller corrector scheduler exponents a and b (ablation) | Discrete Flow Matching 1.7B models · Conditional text generation | Llama-2 generative perplexity | Mixed |
| use a cubic scheduler rather than linear, quadratic, or cosine (ablation) | Discrete Flow Matching · CIFAR10 | FID | Helped |
| use MaskGIT confidence-based token prioritization | discrete masked generation · Image generation | generation quality | Mixed |
Surprising findings
- MaskGIT's confidence-based unmasking prioritization helps at very low NFE but introduces bias that degrades quality at higher NFE.
Limitations
- Sampling efficiency still lags continuous flow matching; no KV-cache so throughput is lower than autoregressive models.
- Non-trivial coding results only with oracle length in some settings; overall pass rates remain well below autoregressive models.
Experimental setup
Datasets: OpenWebText Llama-2 data mix CodeLlama code data mix CIFAR10
Benchmarks: HumanEval MBPP CIFAR10
Metrics: generative perplexity entropy Pass@k FID Inception score
Baselines: Autoregressive models Discrete Flows MaskGIT D3PM Llama-2 7B Llama-3 8B
Builds on: Discrete Flows Flow Matching MaskGIT D3PM
Closest papers by meaning
- Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective (ICLR 2025 · similarity 0.91)
- Blockwise Flow Matching: Improving Flow Matching Models For Efficient High-Quality Generation (NeurIPS 2025 · similarity 0.89)
- Categorical Flow Matching on Statistical Manifolds (NeurIPS 2024 · similarity 0.89)
- On the Closed-Form of Flow Matching: Generalization Does Not Arise from Target Stochasticity (NeurIPS 2025 · similarity 0.89)
- Flow matching achieves almost minimax optimal convergence (ICLR 2025 · similarity 0.89)
- Diff2Flow: Training Flow Matching Models via Diffusion Model Alignment (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.