Research map / Image generation / Autoregressive and latent generation
DPLM-2: A Multimodal Diffusion Protein Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that pretrained masked language models (XLM-RoBERTa, 86M to 9.7B) can be reprogrammed into diffusion language models via diffusive adaptation (reparameterized discrete diffusion), and that scaling them improves translation/summarization, enables zero/few-shot instruction following, and shows early reasoning signs.
Problem
Whether diffusion language models can match autoregressive LLMs on general language tasks and scale with data, model size, and task diversity, given prior diffusion LMs fell short in generation quality.
Approach
Connects absorbing discrete diffusion to masked language modeling so that pretrained MLMs can be treated as diffusion LMs and then adapted (generative surgery / diffusive adaptation) with the reparameterized diffusion objective, followed by task-specific or Flan instruction finetuning.
Key results
- Flan-XLM-R-XXL (9.7B) reaches 30.90 zero-shot SacreBLEU on IWSLT14 De-En, about 2.4 below the supervised Transformer baseline of 33.30.
- XLM-R-XXL (9.7B) with RDM reaches 38.57 SacreBLEU on IWSLT14 De-En and 31.54 Rouge-L on Gigaword-10K with length prediction, outperforming the 39-43M Transformer-BASE AR baseline (33.30 and 10.42).
- On GSM8K, accuracy of the diffusion model rises from 5.6% to 12.8% after tuning on distilled chain-of-thought data, while the largest model still fails to show non-trivial reasoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize diffusion adaptation from pretrained XLM-R instead of random initialization | XLM-R-BASE RDM · IWSLT14 De→En | SacreBLEU: 28.79 (29.12) -> 34.10 (35.78) | Helped |
| Initialize diffusion adaptation from pretrained XLM-R instead of random initialization | XLM-R-BASE RDM · Gigaword-10K | ROUGE-L: 10.01 (10.66) -> 27.52 (28.83) | Helped |
| Initialize diffusion adaptation from pretrained XLM-R instead of random initialization | XLM-R-BASE RDM · WMT14 En→De | SacreBLEU: 26.09 (26.86) -> 26.65 (26.64) | Helped |
| Use RDM rather than autoregressive training in a decoder-only XLM-R-BASE | XLM-R-BASE · IWSLT14 De→En | SacreBLEU: 26.07 -> 28.79 (29.12) | Helped |
| Scale pretrained XLM-R RDM from 86M to 9.7B parameters | XLM-R RDM · IWSLT14 De→En | SacreBLEU: 34.10 (35.78) -> 38.57 (40.65) | Helped |
| Scale pretrained XLM-R RDM from 86M to 9.7B parameters | XLM-R RDM · WMT14 En→De | SacreBLEU: 26.65 (26.64) -> 30.34 (32.81) | Helped |
| Scale pretrained XLM-R RDM from 86M to 9.7B parameters | XLM-R RDM · Gigaword-10K | ROUGE-L: 27.52 (28.83) -> 31.54 (32.57) | Helped |
| Scale instruction-tuned XLM-R from 85M to 9.7B parameters | Flan-XLM-R · IWSLT14 De→En, Strict Flan'21 | SacreBLEU: 7.15 -> 28.74 | Helped |
| Scale instruction-tuned XLM-R from 85M to 9.7B parameters | Flan-XLM-R · IWSLT14 De→En, Flan'22 | SacreBLEU: 21.26 -> 29.59 | Helped |
| Add GSM8K chain-of-thought tuning data distilled from code-davinci-002 | Flan-XLM-R · GSM8K | accuracy: 5.6% -> 12.8% | Helped |
Surprising findings
- Diffusion LMs generate content in causal orders without specific tuning, suggesting planning or backtracking-like behavior.
- Decoder-only diffusion with XLM-R outperforms autoregressive decoder-only models trained from scratch on IWSLT14 by a large margin.
Limitations
- Built by adapting existing MLMs (XLM-R, released years earlier) rather than pretraining from scratch, so architecture and data lag state-of-the-art LLMs.
- Maximum length of 512 tokens and weak arithmetic ability.
- Reasoning remains weak: the largest model fails on GSM8K and MGSM.
Experimental setup
Datasets: IWSLT14 De-En WMT14 En-De Gigaword-10K Flan 2021 Collection Flan 2022 Collection CC100 GSM8K chain-of-thought (distilled from code-davinci-002)
Benchmarks: IWSLT14 De-En WMT14 En-De Gigaword-10K MMLU BBH-NLP TyDiQA GSM8K MGSM
Metrics: SacreBLEU Rouge-L accuracy
Baselines: Transformer-BASE (autoregressive) Transformer-BASE (RDM) mT5 T5 Flan-T5 DiffusionBERT
Builds on: reparameterized discrete diffusion (RDM) XLM-RoBERTa Flan 2022 Collection DiffusionBERT
Closest papers by meaning
- Diffusion Language Models Are Versatile Protein Learners (ICML 2024 · similarity 0.95)
- Diffusion on Language Model Encodings for Protein Sequence Generation (ICML 2025 · similarity 0.92)
- Elucidating the Design Space of Multimodal Protein Language Models (ICML 2025 · similarity 0.91)
- DePLM: Denoising Protein Language Models for Property Optimization (NeurIPS 2024 · similarity 0.90)
- Structure Language Models for Protein Conformation Generation (ICLR 2025 · similarity 0.89)
- Understanding protein function with a multimodal retrieval-augmented foundation model (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.