Research map / Large language models / Reasoning and chain of thought
InfoDLM: an Information-Adaptive Framework for Discrete Diffusion Language Model Pretraining
Shirou Jing, Chunshu Wu, Chuan Liu, Arghavan Bahadorinejad, Feitong Qiao, Dongfang Liu, Tony Geng
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfoDLM, an information-adaptive masking framework for discrete diffusion language model pretraining that targets informative tokens instead of heuristic random masking.
Problem
Discrete diffusion language models (DLMs) are pretrained with heuristic random masking, which does not focus on the most informative tokens, so compute is wasted on redundant or trivial tokens.
Approach
Replaces random masking in mask-and-denoise DLM pretraining with an adaptive, information-based masking scheme that selects which tokens to mask according to their informativeness. Full method details are unavailable beyond the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace random masking with a policy trained on the TIG reward (ablation) | SMDM 472M · GSM8K, 20B tokens | accuracy: 40.29 -> 50.8 | Improved |
| add cyclic (alternating) policy/DLM training without phase conditioning (ablation) | SMDM + TIG reward · GSM8K, 35B tokens | accuracy: 51.07 -> 50.61 | No clear effect |
| add phase signal via FiLM conditioning of mask policy (ablation) | SMDM + TIG reward + cyclic training · GSM8K, 20B tokens | accuracy: 48.52 -> 51.78 | Improved |
| add phase signal via FiLM conditioning of mask policy (ablation) | SMDM + TIG reward + cyclic training · ANLI, 35B tokens | accuracy: 46.70 -> 49.30 | Improved |
| use learned InfoDLM masking instead of random masking (ablation) | SMDM 472M · ANLI, 35B tokens | accuracy: 35.50 -> 49.30 | Improved |
| use span masking instead of random masking (ablation) | SMDM 472M · GSM8K, 35B tokens | accuracy: 40.48 -> 41.42 | No clear effect |
| use PMI or frequency heuristic masking instead of random masking (ablation) | SMDM 472M · GSM8K, 35B tokens | accuracy: 40.48 -> 42.54 (PMI) | Improved |
| InfoDLM at 10B tokens versus baselines at 35B tokens (ablation) | InfoDLM 472M · ANLI | accuracy: 41.20 vs 39.60 (PMI, 35B) | Improved |
| InfoDLM instead of SMDM, zero-shot HellaSwag at 472M | 472M model · HellaSwag (zero-shot) | accuracy: 30.04 -> 27.99 | Hurt |
| InfoDLM instead of SMDM at 1B scale, GSM8K SFT | 1B model · GSM8K (SFT) | accuracy: 57.09 -> 57.24 | No clear effect |
| InfoDLM instead of SMDM at 1B scale, zero-shot | 1B model · HellaSwag (zero-shot) | accuracy: 36.48 -> 42.37 | Improved |
| alternating MPN updates add compute overhead | InfoDLM 472M · per 5:1 cycle | time per cycle (s): 10.10 -> 11.52 | Hurt |
Closest papers by meaning
- From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.92)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.92)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.91)
- On the Reasoning Abilities of Masked Diffusion Language Models (ICLR 2026 · similarity 0.91)
- Looped Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
- SemanticDLM+: Improving Diffusion LLMs through Bias-variance Trade-off in Transition Kernel Design (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
