Research map / Probabilistic and scientific modeling / Molecules and proteins
SaProt: Protein Language Modeling with Structure-aware Vocabulary
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a structure-aware vocabulary that fuses residue tokens with Foldseek 3Di structure tokens and pretrains an ESM-2-650M-sized protein language model (SaProt) on ~40M AlphaFold structures.
Problem
Vanilla protein language models are trained only on residue sequences and lack explicit protein structure information, while structure-based models are limited to small sets of experimental or predicted structures and are not general-purpose sequence-style PLMs.
Approach
Combine each residue token with a Foldseek 3Di structure token into a 441-token structure-aware (SA) vocabulary, so proteins become sequences usable by standard BERT/ESM-style MLM models, and pretrain on ~40M AlphaFold structures with a masking strategy that predicts only the residue part while keeping structure tokens visible.
Key results
- SaProt achieves 0.909 AUC on ClinVar vs 0.862 for ESM-2 and 0.478 vs 0.475 Spearman on ProteinGym without MSA retrieval.
- SaProt reaches 86.41% accuracy on Thermostability-related and 0.724 Spearman on Thermostability vs 0.680 for ESM-2, and 85.57% on DeepLoc subcellular vs 82.09% for ESM-2.
- On contact prediction (TAPE), SaProt improves long-range P@L from 39.16 (ESM-2) to 77.16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use structure-aware residue-plus-structure tokens instead of residue-only input | SaProt (ESM-2 650M backbone) · ClinVar | AUC: 0.900 -> 0.909 | Helped |
| Use structure-aware residue-plus-structure tokens instead of residue-only input | SaProt (ESM-2 650M backbone) · ProteinGym (w/o MSA retrieval) | Spearman’s ρ: 0.475 -> 0.478 | Helped |
| Use structure-aware residue-plus-structure tokens instead of residue-only input | SaProt (ESM-2 650M backbone) · ProteinGym (w/ MSA retrieval) | Spearman’s ρ: 0.479 -> 0.489 | Helped |
| Use structure-aware residue-plus-structure tokens instead of residue-only input | SaProt (ESM-2 650M backbone) · Thermostability | Spearman’s ρ: 0.680 -> 0.724 | Helped |
| Add SaProt in place of ESM-2 in the ESM-GearNet ensemble | ESM-GearNet · HumanPPI | ACC%: 82.30 -> 84.16 | Helped |
| Add SaProt in place of ESM-2 in the ESM-GearNet ensemble | ESM-GearNet · GO BP | Fmax: 0.372 -> 0.381 | Helped |
| Add SaProt in place of ESM-2 in the ESM-GearNet ensemble | ESM-GearNet · GO CC | Fmax: 0.424 -> 0.435 | Helped |
| Combine SaProt with GearNet rather than using SaProt alone | SaProt · GO BP | Fmax: 0.356 -> 0.381 | Helped |
| Combine SaProt with GearNet rather than using SaProt alone | SaProt · Thermostability | Spearman’s ρ: 0.724 -> 0.660 | Hurt |
| Predict the residue token while revealing the structure token, rather than predicting both tokens (ablation) | SaProt pre-training · ProteinGym (w/o MSA retrieval) | Spearman’s ρ: 0.474 -> 0.478 | Helped |
| Predict the residue token while revealing the structure token, rather than predicting both tokens (ablation) | SaProt pre-training · DeepLoc Subcellular | Acc%: 83.65 -> 85.57 | Helped |
| Replace structure tokens with random Foldseek tokens (ablation) | SaProt · ProteinGym | Spearman’s ρ: 0.478 -> 0.452 | Hurt |
Surprising findings
- Training directly on predicted AlphaFold structures with GNN or structure-bias models gave poor validation performance on real PDB structures, suggesting overfitting to AF2 artifacts.
- ESM-2 15B did not outperform the 650M version on zero-shot tasks, and SaProt outperformed the 15B model.
Limitations
- Performance depends heavily on the accuracy of Foldseek structure tokens and AF2 predictions.
- Model size may not have reached maximum capacity due to computational constraints.
- Residue-only fine-tuning shows no clear advantage over ESM models.
Experimental setup
Datasets: AlphaFoldDB structures (~40M) ProteinGym ClinVar DeepLoc Thermostability (FLIP) Metal Ion Binding EC and GO (DeepFRI) HumanPPI (PEER) TAPE tasks AAV (FLIP) beta-lactamase (PEER) SCOPe
Benchmarks: ClinVar ProteinGym Thermostability Metal Ion Binding DeepLoc EC GO HumanPPI TAPE contact prediction AAV beta-lactamase
Metrics: AUC Spearman's rho accuracy Fmax P@L
Baselines: ESM-2 ESM-1b ESM-1v Tranception L ESM-IF MIF-ST EVE GearNet ProstT5
Builds on: ESM-2 Foldseek BERT ESM-GearNet
Closest papers by meaning
- Elucidating the Design Space of Multimodal Protein Language Models (ICML 2025 · similarity 0.90)
- From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models (ICML 2025 · similarity 0.89)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.88)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.88)
- Protein Structure Tokenization: Benchmarking and New Recipe (ICML 2025 · similarity 0.88)
- DPLM-2: A Multimodal Diffusion Protein Language Model (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.