Research map / Image generation / Fast diffusion sampling

Extracting Training Data from Diffusion Language Models via Infilling

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces infilling extraction, a mask-parameterized training-data extraction protocol for diffusion language models, showing edge-conditioned and targeted infilling leak far more than prefix-only probing, including redacted PII.

Problem

Training-data extraction in diffusion language models has been evaluated only with prefix-conditioned probing inherited from autoregressive models, which likely underestimates DLM leakage risk given their bidirectional denoising.

Approach

Generalize prefix-conditioned extraction to arbitrary binary masks (infilling extraction), defining five access modes (prefix, suffix, edge, middle, random) plus a targeted PII-only mask, and evaluate across FT, LoRA, and SFT pipelines on LLaDA-8B and Dream-7B.

Key results

• Edge-conditioned masks extract up to ~3x more verbatim sequences than prefix; LLaDA-8B-FT edge ASR 0.41% vs prefix 0.18% on MIMIR Pile.
• Targeted infilling on redacted Enron data reaches 40.5% exact-match recall for emails on LLaDA-8B-FT, exceeding LLaMA-2-7B dynamic-prefix recall of 35.9%.
• Prompts are recoverable from responses on Tulu3: prompt ASR 10.32% (LLaDA-SFT) and response-conditioned ASR 3.36%.

Surprising findings

• Under random masking, token-level recovery (TER ~56%) is high while verbatim ASR stays below 0.4%, so TER is a poor proxy for verbatim extraction.
• Targeted infilling lets DLMs match or exceed a scale-matched ARM on email recall even though DLMs leak less under prefix-only extraction.

Limitations

• Memorization is induced by fine-tuning rather than measured on pre-training data of LLaDA and Dream.
• Only three exposure regimes studied; preference-based post-training such as RLHF not considered.
• DLM vs ARM comparison is scale-matched and tuning-matched only, not a strict comparison.

Experimental setup

Datasets: MIMIR Pile Enron Email Tulu3

Benchmarks: MIMIR Pile Enron Email Tulu3

Metrics: ASR TER recall precision recall_k

Baselines: LLaMA-2-7B (dynamic prefix) prefix-conditioned extraction

Builds on: diffusion language models prefix-conditioned extraction LoRA Presidio

Closest papers by meaning

• Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects (NeurIPS 2026 · similarity 0.93)
• Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.90)
• Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.89)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.89)
• Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.88)
• Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.88)