Research map / Image generation / Fast diffusion sampling
Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes discoverable (n,p) extraction to masked diffusion LMs, proves sampling resolution monotonically increases exact memorization, and finds DLMs leak less PII than size-matched ARMs.
Problem
Memorization and verbatim extraction of training data in diffusion language models is unexplored; existing discoverable-extraction definitions assume prefix-suffix autoregressive decoding and do not cover bidirectional masked denoising.
Approach
A generalized (n,p)-discoverable extraction definition over arbitrary masks and stochastic sampling trajectories, with a theorem that finer sampling resolution (more denoising steps) weakly increases exact-recovery probability, so ARM decoding is the limiting case of maximal resolution.
Key results
- LLaDA-8B memorized email samples at p=50% rise from 9 (one-step) to 179 (max-step) with n=10,000 on 3,000 prompts, comparable to ARM-1.1B's 213.
- DLM-1.1B yields 16 memorized emails at p=50% under max-step, versus 213 for the scale-matched ARM-1.1B.
- Empirical exact-recovery rate increases approximately monotonically with the number of denoising steps across 200 mask patterns.
Surprising findings
- Under the aligned setup, the 8B diffusion model with per-token reconstruction is only comparable to the 1.1B ARM despite roughly 150x more training tokens.
- Phone-number memorization is essentially zero for all DLM scales even at high query budgets.
Limitations
- Monotonicity theorem relies on an assumption that correct context never reduces recovery probability, which is only empirically supported.
- Experiments use a limited subset of 200 mask patterns for the resolution analysis due to compute.
- ARM and DLM comparison confounded by training-objective differences and LLaDA-8B training FLOPs.
Experimental setup
Datasets: SlimPajama Enron email dataset TREC 2007 Spam
Benchmarks: Enron email PII extraction (3,000 prompts) Enron phone PII extraction (3,000 prompts) Enron vs TREC 2007 Spam membership contrast
Metrics: (n,p)-discoverable extraction counts empirical vs theoretical memorization probability log success-rate ratio vs one-step
Baselines: ARM-1.1B (same setup) LLaDA-8B autoregressive-style one-step reconstruction
Builds on: discoverable extraction (Carlini et al.) masked diffusion language models LLaDA
Closest papers by meaning
- Extracting Training Data from Diffusion Language Models via Infilling (NeurIPS 2026 · similarity 0.93)
- Detecting, Explaining, and Mitigating Memorization in Diffusion Models (ICLR 2024 · similarity 0.91)
- On the Edge of Memorization in Diffusion Models (NeurIPS 2025 · similarity 0.91)
- How Diffusion Models Memorize (NeurIPS 2026 · similarity 0.90)
- Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.90)
- On the Memorization of Consistency Distillation for Diffusion Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.