Research map / Image generation / Autoregressive and latent generation
Diffusion on Language Model Encodings for Protein Sequence Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiMA, a continuous latent diffusion model that denoises protein language model embeddings (ESM-2, ESMc, CHEAP, SaProt) and decodes them to sequences, evaluated on unconditional generation and several conditional protein design tasks.
Problem
Continuous diffusion for protein sequence generation has been underexplored; prior continuous approaches were limited to specific representations or conditional tasks, and discrete diffusion/autoregressive models suffer from repetition and low diversity.
Approach
Run Gaussian latent diffusion on normalized pLM encoder representations with a tan-based noise schedule (tan-10), self-conditioning, long skip connections, fine-tuned decoders, and attention-masked length handling; the architecture is encoder-agnostic and reused across ESM-2, ESMc, CHEAP and SaProt encodings.
Key results
- On SwissProt with ESM-8M encoder, DiMA (35M) achieves FD-seq 0.34 and pLDDT 83.3, outperforming baselines such as DPLM (0.50, 84.0 pLDDT but novelty 11.5 vs 35.7) and nanoGPT (1.24).
- Scaling the ESM-2 encoder from 8M to 3B raises pLDDT from 74.25 to 83.40 and lowers FD-seq from 0.560 to 0.314 on AFDBv4-90.
- DiMA with SaProt-650M solves 19 of 24 motif-scaffolding problems; fold-conditioned generation on CATH hold-out reaches mean TM-score 0.93 with 100% success versus RFDiffusion's 0.48 and 41%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiMA latent diffusion instead of the strongest reported FD-seq baseline | Protein generation on SwissProt · SwissProt | FD-seq: 1.19 -> 0.34 | Helped |
| Use DiMA rather than Walk-Jump | Protein generation on SwissProt · SwissProt | CD₀.₅: 1.000 -> 0.611 | Hurt |
| Add long skip connections (ablation) | DiMA · SwissProt | FD-seq: 0.45 -> 0.34 | Helped |
| Use dedicated time layers instead of admixing time embeddings into corrupted latents (ablation) | DiMA · SwissProt | FD-seq: 0.41 -> 0.34 | Helped |
| Add the ESM-2 encoder (ablation) | DiMA · SwissProt | FD-seq: 1.07 -> 0.34 | Helped |
| Add self-conditioning (ablation) | DiMA · SwissProt | FD-seq: 0.55 -> 0.34 | Helped |
| Fine-tune the decoder (ablation) | DiMA · SwissProt | FD-seq: 0.54 -> 0.34 | Helped |
| Use length sampling (ablation) | DiMA · SwissProt | FD-seq: 0.67 -> 0.34 | Helped |
| Use tan-10 instead of a linear noise schedule (ablation) | DiMA · SwissProt | FD-seq: 0.47 -> 0.34 | Helped |
| Use tan-10 instead of a cosine noise schedule (ablation) | DiMA · SwissProt | FD-seq: 0.94 -> 0.34 | Helped |
| Use diffusion instead of flow matching (ablation) | DiMA · SwissProt | FD-seq: 0.71 -> 0.34 | Helped |
| Scale the encoder from ESM-2 8M to ESM-2 3B | DiMA on AFDBv4-90 · AFDBv4-90 | pLDDT: 74.25 -> 83.40 | Helped |
Surprising findings
- Flow matching gave the highest CD0.5 and novelty in the ablation but much worse pLDDT (63.4) and FD-seq (0.71) than the tan-10 Gaussian diffusion.
- Smaller ESM-2 encoders generate more repetitive sequences, and the encoder is dropped at inference time.
Limitations
- Evaluation of structure relies on ESMFold predictions and pLDDT rather than experimental validation.
- Larger encoders trade off sequence diversity (CD0.5 declines) for quality.
Experimental setup
Datasets: SwissProt AFDBv4-90 CATH S40
Benchmarks: SwissProt AFDBv4-90 RFDiffusion motif scaffolding benchmark CATH S40 hold-out
Metrics: FD-seq pLDDT CD0.5 novelty TM-score scRMSD optimal transport distance MMD success rate
Baselines: RITA nanoGPT DPLM EvoDiff-OADM D3PM DFM Walk-Jump ProteinGAN SeqDesign RFDiffusion
Builds on: self-conditioning EvoDiff DPLM CHEAP
Closest papers by meaning
- Diffusion Language Models Are Versatile Protein Learners (ICML 2024 · similarity 0.93)
- DPLM-2: A Multimodal Diffusion Protein Language Model (ICLR 2025 · similarity 0.92)
- Structure Language Models for Protein Conformation Generation (ICLR 2025 · similarity 0.91)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.90)
- Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction (ICLR 2025 · similarity 0.89)
- Steering Protein Language Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.