Research map / Image generation / Autoregressive and latent generation
MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MSAGPT, a 2.8B-parameter MSA generative pre-trained model with 2D evolutionary positional encoding and 1D zero/few-shot decoding, refined by rejective fine-tuning and DPO with AlphaFold2 feedback to generate virtual MSAs that boost structure prediction in low-MSA regimes.
Problem
Protein structure prediction accuracy drops for "orphan" proteins with too few homologous sequences to build a high-quality MSA.
Approach
A 2D evolutionary positional encoding (column and row position ids via 2D RoPE) lets a standard 1D autoregressive transformer model row- and column-wise co-evolution in a flattened MSA, followed by rejective fine-tuning on AF2-filtered MSAs and DPO using AF2 accuracy as the preference signal.
Key results
- On the natural MSA-scarce benchmark, MSAGPT-DPO reaches 57.0 TM-Score on CASP few-shot versus 44.1 for AF2 MSA (+8.5 over MSAGPT's base, as reported in the table), with gains of +1.4 on CAMEO and +4.7 on PDB over the strongest baseline.
- In zero-shot evaluation on the artificial MSA-scarce benchmark, MSAGPT reaches 50.7 TM and 51.6 TM after RFT and DPO, above EvoGen's 44.6.
- Transfer to four protein tasks with virtual MSA improves contact prediction, secondary structure, and metal ion binding but lowers localization prediction (58.3 to 56.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RFT and DPO feedback training (ablation) | MSAGPT · CASP few-shot | TM-Score: +8.5 | Helped |
| Add RFT and DPO feedback training (ablation) | MSAGPT · CAMEO zero-shot | TM-Score: +2.4 | Helped |
| Add RFT and DPO feedback training (ablation) | MSAGPT · PDB few-shot | TM-Score: +4.7 | Mixed |
| Add RFT and DPO feedback training (ablation) | MSAGPT · Artificial MSA-scarce benchmark, zero-shot | TM: 50.7 -> 51.6 | Helped |
| Select generated MSA using pLDDT (ablation) | MSAGPT-DPO · CAMEO few-shot | TM-Score: 76.7 -> 77.5 | Helped |
| Select generated MSA using pLDDT (ablation) | MSAGPT-DPO · CASP few-shot | TM-Score: 57.0 -> 57.6 | Helped |
| Select generated MSA using pLDDT (ablation) | MSAGPT-DPO · PDB few-shot | TM-Score: 59.7 -> 60.5 | Helped |
| Add virtual MSA | Protein prediction models · Contact Prediction (CtP) | Top (L/5): 11.6 -> 13.1 | Helped |
| Add virtual MSA | Protein prediction models · Secondary Structure Prediction (SsP) | ACC: 66.5 -> 69.0 | Helped |
| Add virtual MSA | Protein prediction models · Localization Prediction (LocP) | ACC: 58.3 -> 56.4 | Hurt |
| Add virtual MSA | Protein prediction models · Metal Ion Binding (MIB) | ACC: 57.5 -> 60.3 | Helped |
| Add generated MSA to 128 natural MSA sequences | AF2 · MSA-abundant CAMEO | TM: 85.1 -> 85.3 | Helped |
Surprising findings
- Adding more virtual MSA beyond a threshold (64 sequences) underperformed 16 or 32 sequences, attributed to a dilution effect.
- RFT and DPO models have higher TM-Score but lower pTM and pLDDT than the base model, indicating confident but less accurate decoys in predictor confidence.
Limitations
- Pre-training only at 2.8B parameters; scaling behavior unknown.
- Transfer learning evaluated on only a few protein tasks.
- Generated MSAs could contaminate natural MSA databases.
Experimental setup
Datasets: OpenProteinSet Uniclust30 MSAs PDB CAMEO CASP14 CASP15 UniClust30
Benchmarks: CAMEO CASP14&15 PDB CtP SsP LocP MIB
Metrics: TM-Score GDT-TS LDDT pTM pLDDT
Baselines: AF2 MSA EvoDiff MSA-Augmentor EvoGen
Builds on: RoPE DPO AlphaFold2 EvoGen MSA Transformer
Closest papers by meaning
- MSA Generation with Seqs2Seqs Pretraining: Advancing Protein Structure Predictions (NeurIPS 2024 · similarity 0.90)
- Protein Multimer Structure Prediction via Prompt Learning (ICLR 2024 · similarity 0.86)
- Learning to Predict Mutational Effects of Protein-Protein Interactions by Microenvironment-aware Hierarchical Prompt Learning (ICML 2024 · similarity 0.86)
- Evolution-Inspired Loss Functions for Protein Representation Learning (ICML 2024 · similarity 0.86)
- Ultrafast classical phylogenetic method beats large protein language models on variant effect prediction (NeurIPS 2024 · similarity 0.86)
- Proteina: Scaling Flow-based Protein Structure Generative Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.