Research map / Image generation / Fast diffusion sampling
Soft-Masked Diffusion Language Models
Michael Hersche, Samuel Moor-Smith, Thomas Hofmann, Abbas Rahimi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add soft-masking feedback (train from scratch) | MDLM 169M, standard unmasking · OpenWebText, NFE 1/1 | MAUVE: 0.034 -> 0.602 | Improved |
| add soft-masking feedback (train from scratch) | MDLM 169M, standard unmasking · OpenWebText, NFE 1/1 | generative perplexity: 50.46 -> 23.53 | Improved |
| add soft-masking on top of ReMDM remasking | MDLM 169M with ReMDM · OpenWebText, NFE 1/1 | MAUVE: 0.411 -> 0.774 | Improved |
| add soft-masking via 100k-step continued pretraining | pretrained binary MDLM · OpenWebText validation | validation perplexity: 23.14 -> 21.63 | Improved |
| use iso-compute (half the update steps) instead of iso-update training for soft-masking (ablation) | MDLM 169M SM · OpenWebText, NFE 1/4 | MAUVE: 0.383 -> 0.417 | No clear effect |
| add soft-masking finetuning | Dream-7B · MBPP+ at NFE 1/2 | accuracy (%): 39.6 -> 54.7 | Improved |
| add soft-masking finetuning | Dream-Coder-7B · MBPP at NFE 1/4 | accuracy (%): 25.9 -> 33.2 | Improved |
| add soft-masking finetuning at full NFE budget | Dream-7B · HumanEval, HumanEval+, MBPP, MBPP+ at NFE 1/1 | accuracy (%): HumanEval 59.5 -> 57.8; HumanEval+ 53.0 -> 50.0 | Hurt |
| add soft-masking finetuning at full NFE budget | Dream-Coder-7B · coding benchmarks at NFE 1/1 | accuracy (%): HumanEval 75.7 -> 76.2 | No clear effect |
| use top-k=1 versus k=3 superposition (ablation) | soft-masking · coding vs language modeling | accuracy / perplexity | Mixed |
| use softmax with trainable temperature instead of top-k (ablation) | soft-masking · language modeling and coding | validation perplexity / accuracy | No clear effect |
| apply soft-masking only in early decoding steps (ablation) | soft-masking · decoding schedule | performance | Improved |
Closest papers by meaning
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.94)
- Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.92)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.92)
- Re-evaluating Confidence Remasking in Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.91)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.91)
- Simplified and Generalized Masked Diffusion for Discrete Data (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
