Research map / Probabilistic and scientific modeling / Molecules and proteins
Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training
Jaeyeon Kim, Jonathan Geuter, David Alvarez-Melis, Sham M. Kakade, Sitan Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace random masking with PUMA progressive unmasking forward process | 125M MDM from scratch · TinyGSM, GSM8K test accuracy | iterations to reach accuracy: 2.3x faster | Improved |
| Add PUMA on top of autoregressive initialization | 125M MDM (ARM init) · TinyGSM, GSM8K test accuracy | iterations to reach accuracy: approximately 4.0x speedup | Improved |
| Add PUMA to block diffusion training (block size 256) | Block diffusion 125M · TinyGSM, GSM8K test accuracy | accuracy vs iterations | Improved |
| Add PUMA to LoRA fine-tuning of Dream-Coder-Base-7B | Dream-Coder-Base-7B · HumanEval | Pass@1: about +10 percentage points (vanilla at most +3%) | Improved |
| Add PUMA to LoRA fine-tuning of Dream-Coder-Base-7B | Dream-Coder-Base-7B · MBPP | Pass@1: approximately +5 percentage points | Improved |
| Add PUMA on Sudoku | 6.8M bidirectional MDM · Sudoku | training speedup: 1.7x | Improved |
| Add K-scheduling (K=42 down to 12) rather than fixed K=12 (ablation) | PUMA 125M · TinyGSM, GSM8K test accuracy | training speedup | Improved |
| Lower confidence threshold to 0.4 or 0.65 (default 0.9) (ablation) | PUMA 125M · TinyGSM, GSM8K test accuracy | training efficiency | Hurt |
| Vary confidence threshold among 0.75 to 0.95 (ablation) | PUMA 125M · TinyGSM, GSM8K test accuracy | training efficiency | No clear effect |
| Change inference unmasking policy (Top-K margin, entropy, etc.) for a PUMA-trained model (ablation) | PUMA 125M · TinyGSM, GSM8K test accuracy | accuracy: 2.2x (Top-K margin), 2.3x (entropy) | No clear effect |
| Add PAPL loss reweighting (ablation) | MDM baseline and PUMA · Sudoku and TinyGSM | accuracy | Hurt |
| Use PUMA instead of random masking (wall-clock) | 125M MDM · TinyGSM | training throughput (it/s): 3.78 -> 3.41 | Hurt |
Closest papers by meaning
- Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking (NeurIPS 2025 · similarity 0.93)
- Learn from Your Mistakes: Self-Correcting Masked Diffusion Models (NeurIPS 2026 · similarity 0.91)
- Simplified and Generalized Masked Diffusion for Discrete Data (NeurIPS 2024 · similarity 0.91)
- Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions (ICML 2025 · similarity 0.90)
- Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking (NeurIPS 2025 · similarity 0.90)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
