Research map / Image generation / Fast diffusion sampling
Non-Markovian Discrete Diffusion with Causal Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CaDDi, a non-Markovian discrete diffusion model in a single causal transformer with 2D RoPE that conditions each denoising step on the full generative trajectory, and CaDDi-AR, a token-level variant that can be fine-tuned from pretrained LLMs.
Problem
Discrete diffusion models lag behind causal language models in quality, partly because the Markovian assumption forces each denoising step to depend only on the current state, so early errors accumulate and cannot be corrected.
Approach
Replace the Markov reverse process with a non-Markovian one where each denoising step conditions on the entire trajectory x_{t:T}, using independent noising of x_0 at each step; a decoder-only causal transformer models both token order and diffusion time, with an added temporal rotary encoding (2D RoPE), so standard causal LMs are a special case (T=1) and can be fine-tuned directly.
Key results
- On LM1B at T=0.5 under GPT-2 oracle, CaDDi reaches 45.96 Gen PPL versus 81.44 for the best baseline SEDD (with the Llama-2 oracle, CaDDi reaches 23.81 at T=0.5 vs 46.91 for SEDD).
- On text8 at 64 steps, CaDDi achieves BPD <=1.41, beating the 64-step D3PM (1.51), SEDD (1.46), UDLM (1.60) and MDLM (1.46) variants.
- Fine-tuned Qwen2-1.5B with CaDDi-AR reaches 34.2% on ARC-Challenge (vs 33.7 for base Qwen2) and 66.3% on LAMBADA (vs 63.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CaDDi-AR fine-tuning instead of the base QWen2-1.5B model | QWen2-1.5B · ARC-Challenge | Accuracy (%): 33.7 -> 34.2 | Helped |
| Use CaDDi-AR fine-tuning instead of the base QWen2-1.5B model | QWen2-1.5B · LAMBADA | Accuracy (%): 63.9 -> 66.3 | Helped |
| Use CaDDi-AR fine-tuning instead of the base QWen2-1.5B model | QWen2-1.5B · BoolQ | Accuracy (%): 72.6 -> 71.6 | Hurt |
| Apply classifier-free guidance at γ=1.25 rather than γ=1 (ablation) | CaDDi-CFG · Amazon Polarity, positive condition | Sentiment Accuracy (%): 71.37 -> 73.61 | Helped |
| Apply classifier-free guidance at γ=1.25 rather than γ=1 (ablation) | CaDDi-CFG · Amazon Polarity, negative condition | Sentiment Accuracy (%): 85.42 -> 85.92 | Helped |
| Use CaDDi-AR rather than CaDDi | CaDDi · LM1B, GPT-2 oracle, T=1 | Generative PPL: 142.51 -> 139.80 | Helped |
| Use CaDDi-AR rather than CaDDi | CaDDi · LM1B, Llama-2-7B oracle, T=0.5 | Generative PPL: 23.81 -> 27.76 | Hurt |
| Use CaDDi rather than the strongest reported diffusion baseline at 64 steps | Text8 diffusion models · Text8 test, 64 steps | BPD: ≤1.46 -> ≤1.41 | Helped |
| Use CaDDi rather than the strongest reported diffusion baseline at 64 steps | Text8 diffusion models · Text8 test, 64 steps | Perplexity: ≤2.75 -> ≤2.66 | Helped |
| Use CaDDi rather than the strongest reported diffusion baseline at 64 steps | Text8 diffusion models · Text8 test, 64 steps | NLL: ≤1.01 -> ≤0.98 | Helped |
| Condition denoising on the full trajectory instead of using a Markovian diffusion baseline | CaDDi · Generation with manually injected noise | Generation quality | Helped |
Surprising findings
- At low temperature, the block-level CaDDi outperforms the token-level CaDDi-AR on LM1B generative perplexity, contrary to the expectation that the finer-grained variant is stronger.
Limitations
- CaDDi-AR requires O(L x T) function evaluations, making naive generation slow; the paper relies on semi-speculative decoding to mitigate this.
- BPD is not reported for CaDDi-AR because token-level autoregressive decomposition adds likelihood approximation error.
- Error bars are not reported due to computational cost.
Experimental setup
Datasets: LM1B text8 ARC-Challenge ARC-Easy BoolQ PIQA RACE Social IQA LAMBADA Amazon Polarity
Benchmarks: LM1B text8 ARC-Challenge ARC-Easy BoolQ PIQA RACE Social IQA LAMBADA Amazon Polarity
Metrics: generative perplexity entropy bits-per-dimension accuracy sentiment accuracy
Baselines: D3PM SEDD MDLM UDLM Discrete Flow Matching MDM GPT-2 TinyLlama Qwen2-1.5B
Builds on: D3PM MDLM UDLM DART speculative decoding
Closest papers by meaning
- Constrained Discrete Diffusion (NeurIPS 2025 · similarity 0.90)
- Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time (NeurIPS 2024 · similarity 0.90)
- Continuous Latent Diffusion Language Model (NeurIPS 2026 · similarity 0.89)
- Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.89)
- From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.89)
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.