Research map / Image generation / Fast diffusion sampling
Meta-Diffu$B$: A Contextualized Sequence-to-Sequence Text Diffusion Model with Meta-Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Meta-Diffu$B$, which trains an LSTM scheduler with meta-exploration policy gradients to schedule contextualized diffusion noise for a DiffuSeq-style exploiter on Seq2Seq text generation.
Problem
Existing sequence-to-sequence diffusion models (S2S-Diffusion such as DiffuSeq) use fixed or hand-crafted, non-contextualized noise schedules that ignore the semantics of each conditioning sentence and training epoch.
Approach
A scheduler Seq2Seq model reads the conditioning sentence and emits binary "skip" meta-instructions that modulate a fixed sqrt noise schedule per sentence; it is trained by policy gradient with a meta-reward equal to the BLEU gain of the exploiter after an update, so the noise schedule is learned and contextualized.
Key results
- On QQP, Meta-Diffu$B$ reaches BLEU 0.2632 vs DiffuSeq 0.2413 and SeqDiffuSeq 0.2434, with the lowest Mean-Rank (1.00) across four datasets.
- On WA, Meta-Diffu$B$ achieves BLEU 0.3877 vs DiffuSeq 0.3622; on CC, BLEU 0.0220 vs DiffuSeq 0.0139.
- On IWSLT14 DE-EN, Meta-Diffu$B$ (DiffuSeq exploiter) reports SacreBLEU 31.71 vs DiffuSeq 29.43.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DiffuSeq's fixed noise scheduling with Meta-DiffuB contextualized scheduling | DiffuSeq · QQP | BLEU: 0.2413 -> 0.2632 | Helped |
| replace DiffuSeq's fixed noise scheduling with Meta-DiffuB contextualized scheduling | DiffuSeq · WA | BLEU: 0.3622 -> 0.3877 | Helped |
| replace DiffuSeq's fixed noise scheduling with Meta-DiffuB contextualized scheduling | DiffuSeq · QT | BLEU: 0.1731 -> 0.1820 | Helped |
| replace DiffuSeq's fixed noise scheduling with Meta-DiffuB contextualized scheduling | DiffuSeq · CC | BLEU: 0.0139 -> 0.0220 | Helped |
| apply the pretrained WA scheduler during inference | DiffuSeq trained on QQP · QQP | BLEU: 0.2413 -> 0.2594 | Helped |
| apply the pretrained WA scheduler during inference | DiffuSeq trained on QT · QT | BLEU: 0.1731 -> 0.1804 | Helped |
| apply the pretrained QT scheduler during inference | DiffuSeq trained on WA · WA | BLEU: 0.3622 -> 0.3666 | Helped |
| apply the pretrained WA scheduler during inference | SeqDiffuSeq trained on QQP · QQP | BLEU: 0.2434 -> 0.2627 | Helped |
| apply the pretrained WA scheduler during inference | Dinoiser trained on QQP · QQP | BLEU: 0.1949 -> 0.2079 | Helped |
| schedule less noise for harder sentences and more noise for easier sentences | Meta-DiffuB with DiffuSeq exploiter · WA (hardest sentences) | BLEU | Helped |
Surprising findings
- Plug-and-play use of the scheduler trained on one dataset improves DiffuSeq on other datasets without fine-tuning at inference.
- Larger or more variable noise does not necessarily correlate with better S2S-Diffusion performance.
Limitations
- Scheduler trained with a BLEU-based meta-reward, so the objective is tied to a single lexical metric.
- Gains on some baselines (e.g., Dinoiser, SeqDiffuSeq ablations) are reported with reproduced numbers; results rely on fixed training epochs and MBR settings from prior work.
Experimental setup
Datasets: Commonsense Conversation (CC) Quasar-T (QT) Wiki-Auto (WA) Quora Question Pairs (QQP) IWSLT14 DE-EN WMT14 DE-EN
Benchmarks: CC QT WA QQP IWSLT14 DE-EN WMT14 DE-EN
Metrics: BLEU ROUGE-L BERTScore Dist-1 Self-BLEU Mean-Rank SacreBLEU
Baselines: DiffuSeq SeqDiffuSeq Dinoiser GPT2-base GPT2-large LevT
Builds on: DiffuSeq Diffusion-LM Meta-Exploration Dinoiser SeqDiffuSeq
Closest papers by meaning
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.88)
- Theoretical Benefit and Limitation of Diffusion Language Model (NeurIPS 2025 · similarity 0.88)
- Energy-Based Diffusion Language Models for Text Generation (ICLR 2025 · similarity 0.88)
- Denoising Time Matters: Diverse Generation in Diffusion Language Models (NeurIPS 2026 · similarity 0.88)
- The Diffusion Duality (ICML 2025 · similarity 0.87)
- TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.