Research map / Label-efficient and robust learning / Self-supervised and contrastive
MSA Generation with Seqs2Seqs Pretraining: Advancing Protein Structure Predictions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pretrains MSA-Generator, a 260M encoder-decoder protein language model on a seqs2seqs task (source/target MSA splits from JackHMMER searches), to generate virtual MSAs that boost AlphaFold2 and RoseTTAFold on CASP14/15 low-homolog targets.
Problem
Protein structure prediction with MSA-dependent models (AlphaFold2, RoseTTAFold) degrades when query proteins have few homologous sequences, so search-based MSAs are shallow or poor.
Approach
Frame MSA generation as an unsupervised sequences-to-sequences task: given a source subset of an MSA, generate a target set of co-evolutionary sequences in parallel, using tied-row, cross-row and cross-column attention, and pick the generated MSA with the highest pLDDT over three runs.
Key results
- On CASP14 real-world low-homolog MSAs, AlphaFold2+MSA-Generator raised LDDT from 53.2 to 57.5 and pLDDT from 65.1 to 71.6 (single-sequence and other baselines lower).
- On CASP15, AlphaFold2+MSA-Generator improved LDDT from 55.6 to 63.7 and pLDDT from 65.1 to 73.7.
- On artificial challenging MSAs, generated virtual MSAs improved LDDT for 72.8% of sequences over the 5-sequence baseline, with an average gain of 12.87 LDDT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| augment the searched MSA with MSA-Generator | AlphaFold2 · CASP14 real-world challenging MSAs | LDDT: 53.2 -> 57.5 (+4.3) | Helped |
| augment the searched MSA with MSA-Generator | AlphaFold2 · CASP15 real-world challenging MSAs | LDDT: 55.6 -> 63.7 (+8.1) | Helped |
| augment the searched MSA with MSA-Generator | RoseTTAFold · CASP14 real-world challenging MSAs | LDDT: 51.3 -> 56.3 (+5.0) | Helped |
| augment the searched MSA with MSA-Generator | RoseTTAFold · CASP15 real-world challenging MSAs | LDDT: 52.1 -> 58.4 (+6.3) | Helped |
| augment the searched MSA with MSA-Generator | AlphaFold2 · Orphan25 | LDDT: 61.6 -> 66.4 | Helped |
| replace 5% of MSA tokens with random tokens (ablation) | AlphaFold2 · CASP14 | LDDT: 53.2 -> 50.6 | Hurt |
| replace 10% of MSA tokens with random tokens (ablation) | AlphaFold2 · CASP14 | LDDT: 53.2 -> 49.5 | Hurt |
| replace 15% of MSA tokens with random tokens (ablation) | AlphaFold2 · CASP14 | LDDT: 53.2 -> 44.3 | Hurt |
| replace 5% of MSA tokens with random tokens (ablation) | AlphaFold2 · CASP15 | LDDT: 55.6 -> 50.9 | Hurt |
| replace 10% of MSA tokens with random tokens (ablation) | AlphaFold2 · CASP15 | LDDT: 55.6 -> 43.2 | Hurt |
| replace 15% of MSA tokens with random tokens (ablation) | AlphaFold2 · CASP15 | LDDT: 55.6 -> 39.6 | Hurt |
| increase the number of ensemble runs (ablation) | MSA-Generator with PSP models · CASP14 and CASP15 real-world challenging MSAs | pLDDT | Helped |
Surprising findings
- Randomly substituting 5-15% of MSA tokens with random residues hurt AlphaFold2 performance, while generated MSAs helped.
- Higher pLDDT did not always correspond to higher LDDT when selecting among generated MSAs.
Limitations
- No scale-up experiments on model size or pretraining data.
- Selecting by maximum pLDDT is suboptimal and costs repeated predictions; pLDDT sometimes disagreed with LDDT.
- Generated MSAs do not reach substantial depth, and deeper MSAs increase compute cost.
Experimental setup
Datasets: UniRef90 CASP14 CASP15 Orphan25 UniRef30 ColabFoldDB
Benchmarks: CASP14 CASP15 Orphan25
Metrics: LDDT pLDDT TM-Score GDT-TS Shannon entropy PSSM correlation
Baselines: AlphaFold2 RoseTTAFold ESMFold OmegaFold Potts generation Iterative Unmasking (MSA Transformer)
Builds on: MSA Transformer AlphaFold2 RoseTTAFold Transformer encoder-decoder JackHMMER
Closest papers by meaning
- MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-Training (NeurIPS 2024 · similarity 0.90)
- Diffusion on Language Model Encodings for Protein Sequence Generation (ICML 2025 · similarity 0.86)
- Proteina: Scaling Flow-based Protein Structure Generative Models (ICLR 2025 · similarity 0.86)
- Diffusion Language Models Are Versatile Protein Learners (ICML 2024 · similarity 0.85)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.85)
- SaProt: Protein Language Modeling with Structure-aware Vocabulary (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.