Research map / Image generation / Fast diffusion sampling
Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from $k$-Parity
Jianhao Huang, Baharan Mirzasoleiman
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| restrict mask-rate sampling to t in [0.45, 0.55] during pretraining | LLaDA-8B pretraining from scratch · HellaSwag | accuracy: 0.354 -> 0.400 | Improved |
| restrict mask-rate sampling to t in [0.45, 0.55] during pretraining | LLaDA-8B pretraining from scratch · ARC-Easy | accuracy: 0.342 -> 0.430 | Improved |
| restrict mask-rate sampling to t in [0.45, 0.55] in SFT | LLaDA-8B Base SFT · GPQA | accuracy: 0.344 -> 0.402 | Improved |
| restrict mask-rate sampling to t in [0.45, 0.55] in SFT | LLaDA-8B Base SFT · MMLU, MMLU-stem, ARC-Challenge | accuracy: MMLU 0.659 -> 0.669; ARC-Challenge 0.468 -> 0.480 | Improved |
| restrict mask-rate sampling to the mid-range window [0.45, 0.55] in SFT | LLaDA-8B Base SFT · GSM8K | accuracy: 0.768 -> 0.738 | Hurt |
| restrict mask-rate sampling to the mid-range window [0.45, 0.55] in SFT | LLaDA-8B Base SFT · MATH | accuracy: 0.341 -> 0.342 | No clear effect |
| shift sampling interval toward t=1, window [0.5, 1.0] instead of [0, 1] (ablation) | LLaDA-8B Base SFT · GSM8K | accuracy: 0.768 -> 0.785 | Improved |
| shift sampling interval toward t=1, window [0.5, 1.0] instead of [0, 1] (ablation) | LLaDA-8B Base SFT · MATH | accuracy: 0.341 -> 0.375 | Improved |
| train only on mask rates near the boundaries (t in [0, 0.1] or [0.9, 1.0]) (ablation) | 50M nanoGPT MDLM · WikiText test loss | test loss | Hurt |
| train only on mid-range mask rates t in [0.4, 0.5] or [0.5, 0.6] (ablation) | 50M nanoGPT MDLM · WikiText test loss | test loss: 3.88 -> 3.62 | Improved |
| use masked diffusion objective instead of direct output supervision | nanoGPT on (20,6) parity · k-parity | validation accuracy / grokking | Improved |
| mask-rate range U[0,0.2] or U[0,0.3] rather than U[0,0.1] or U[0,0.4] (ablation) | nanoGPT MD on (20,6) parity · k-parity | convergence speed | Improved |
Closest papers by meaning
- Simplified and Generalized Masked Diffusion for Discrete Data (NeurIPS 2024 · similarity 0.91)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.90)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.90)
- Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
- Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking (NeurIPS 2025 · similarity 0.90)
- UnMaskFork: Test-Time Scaling for Masked Diffusion via Deterministic Action Branching (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
