Research map / Image generation / Autoregressive and latent generation
Diffusion Language Models Are Versatile Protein Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that masked language models (XLM-R, 86M to 9.7B) can be reprogrammed into discrete diffusion language models via diffusive adaptation, then scaled and instruction-tuned (Flan 2022) for translation, summarization, zero-/few-shot and reasoning evaluation.
Problem
Whether diffusion language models can match autoregressive LMs on general language tasks and scale with data, model size, and number of tasks, given that prior diffusion LMs lag in generation quality.
Approach
Exploits the connection between absorbing discrete diffusion and masked language modeling to turn pretrained MLMs into diffusion LMs by finetuning with the reparameterized diffusion objective (diffusive adaptation), then applies task-specific and instruction finetuning.
Key results
- Flan-XLM-R-XXL (9.7B) reaches 30.90 zero-shot SacreBLEU on IWSLT14 De-En, about 2.4 below the supervised Transformer baseline (33.30).
- Fine-tuned XLM-R-XXL (9.7B) achieves 38.57 SacreBLEU on IWSLT14 and 30.34 on WMT14, outperforming the Transformer-BASE baseline (33.30 and 26.85).
- Pretrained XLM-R-BASE with diffusion finetuning gains on Gigaword-10K from 10.01 to 27.52 Rouge-L versus random initialization, showing benefit of MLM pretraining for small data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize diffusion RDM from a pretrained XLM-R-BASE instead of training from scratch | RDM · IWSLT14 De→En | SacreBLEU: 28.79 -> 34.10 | Helped |
| Initialize diffusion RDM from a pretrained XLM-R-BASE instead of training from scratch | RDM · WMT14 En→De | SacreBLEU: 26.09 -> 26.65 | Helped |
| Initialize diffusion RDM from a pretrained XLM-R-BASE instead of training from scratch | RDM · Gigaword-10K | Rouge-L: 10.01 -> 27.52 | Helped |
| Scale the pretrained diffusion model from 86M to 9.7B parameters | XLM-R RDM · IWSLT14 De→En | SacreBLEU: 34.10 -> 38.57 | Helped |
| Scale the pretrained diffusion model from 86M to 9.7B parameters | XLM-R RDM · WMT14 En→De | SacreBLEU: 26.65 -> 30.34 | Helped |
| Scale the pretrained diffusion model from 86M to 9.7B parameters | XLM-R RDM · Gigaword-10K | Rouge-L: 27.52 -> 31.54 | Helped |
| Scale instruction-tuned models from 85M to 9.7B parameters | Flan-XLM-R · IWSLT14 De→En, Strict Flan'21 | SacreBLEU: 7.15 -> 28.74 | Helped |
| Scale instruction-tuned models from 85M to 9.7B parameters | Flan-XLM-R · IWSLT14 De→En, Flan'22 | SacreBLEU: 21.26 -> 29.59 | Helped |
| Add GSM8K chain-of-thought tuning data distilled from code-davinci-002 | Flan-XLM-R-XXL · GSM8K | accuracy: 5.6% -> 12.8% | Helped |
Surprising findings
- Flan-XLM-R-XXL fails to show non-trivial reasoning on GSM8K, though accuracy rises from 5.6% to 12.8% after chain-of-thought tuning on distilled data.
Limitations
- Built by adapting older masked LMs (XLM-R) rather than pretraining diffusion LMs from scratch; architecture and data lag state-of-the-art LLMs.
- Short maximum length (512 tokens) and weak arithmetic/reasoning ability.
- Masking-ratio mismatch between BERT-style pretraining and diffusion may limit how much diffusive adaptation can bridge.
Experimental setup
Datasets: IWSLT14 De-En WMT14 En-De Gigaword-10K Flan 2021 Collection Flan 2022 Collection CC100 GSM8K chain-of-thought from code-davinci-002
Benchmarks: IWSLT14 De-En WMT14 En-De Gigaword-10K MMLU BBH-NLP TyDiQA GSM8K MGSM
Metrics: SacreBLEU Rouge-L accuracy
Baselines: Transformer-BASE mT5 T5 Flan-T5 autoregressive LM from scratch
Builds on: reparameterized discrete diffusion models DiffusionBERT XLM-RoBERTa Flan 2022 collection
Closest papers by meaning
- DPLM-2: A Multimodal Diffusion Protein Language Model (ICLR 2025 · similarity 0.95)
- Diffusion on Language Model Encodings for Protein Sequence Generation (ICML 2025 · similarity 0.93)
- Steering Protein Language Models (ICML 2025 · similarity 0.90)
- DePLM: Denoising Protein Language Models for Property Optimization (NeurIPS 2024 · similarity 0.90)
- Structure Language Models for Protein Conformation Generation (ICLR 2025 · similarity 0.89)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.