Research map / Architectures and efficiency / Attention and transformers
ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ProSST, a protein language model that pairs a GVP-based k-means structure quantizer with sequence-structure disentangled attention, pre-trained with MLM on 18.8M AlphaFold structures.
Problem
Most protein language models use only sequence and ignore structure, while existing structure-aware PLMs use coarse Foldseek tokens and naive self-attention that cannot explicitly model residue-structure relations.
Approach
A GVP local-structure encoder (trained by denoising autoencoding on CATH) is clustered with k-means into K=2048 structure tokens, and a DeBERTa-style disentangled attention computes residue-residue, residue-structure, residue-position and their reverse interactions, dropping the structure-structure and position-position terms.
Key results
- On ProteinGYM zero-shot mutation prediction, ProSST reaches Spearman 0.504 versus 0.457 for the best prior baselines (SaProt, GEMME, Trancepiton-EVE).
- On supervised tasks, ProSST (110M) gets the best result in 4 of 6 settings, e.g. DeepLoc accuracy 94.32% vs 93.55% for SaProt-650M and ESM-GearNet.
- Ablation: K=2048 structure tokens gives the best ProteinGYM Spearman; Foldseek tokens give 0.468 and K=0 (no structure) gives 0.392.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ProSST structure-aware modeling | ProSST versus SaProt · ProteinGYM | Spearman ρs: 0.457 -> 0.504 | Helped |
| add ProSST sequence-structure modeling | protein downstream fine-tuning · DeepLoc | accuracy: 93.55 -> 94.32(±0.10) | Helped |
| add ProSST sequence-structure modeling | protein downstream fine-tuning · Metal Ion Binding | accuracy: 75.75 -> 76.37(±0.02) | Helped |
| add ProSST sequence-structure modeling | protein downstream fine-tuning · Thermostability | Spearman ρs: 0.724 -> 0.726(±0.04) | No clear effect |
| add ProSST sequence-structure modeling | protein downstream fine-tuning · GO-BP | F1-Max: 0.516 -> 0.492(±0.004) | Hurt |
| add ProSST sequence-structure modeling | protein downstream fine-tuning · GO-CC | F1-Max: 0.507 -> 0.501(±0.002) | Hurt |
| use 2048 local-structure tokens instead of no structure tokens (ablation) | ProSST · ProteinGYM | Spearman ρs: 0.392 -> 0.504 | Helped |
| use 2048 local-structure tokens instead of no structure tokens (ablation) | ProSST · DeepLoc | accuracy: 89.77 (±0.26) -> 94.32 (±0.10) | Helped |
| use 2048 local-structure tokens instead of 1024 (ablation) | ProSST · ProteinGYM | Spearman ρs: 0.485 -> 0.504 | Helped |
| replace Foldseek quantization with the proposed quantizer (ablation) | ProSST · ProteinGYM | Spearman ρs: 0.468 -> 0.504 | Helped |
| replace DSSP quantization with the proposed quantizer (ablation) | ProSST · ProteinGYM | Spearman ρs: 0.439 -> 0.504 | Helped |
| add residue-to-structure attention (ablation) | ProSST disentangled attention · ProteinGYM | Spearman ρs: 0.438 -> 0.504 | Helped |
Surprising findings
- Disentangled attention with a constant structure token (K=1) gives almost no gain over K=0, so the attention alone does not help without informative structure tokens.
- Structure-aware model performance correlates positively with AlphaFold pLDDT (rho about 0.30 for ProSST), suggesting limits on disordered proteins.
Limitations
- Local structure construction and encoding is computationally heavy.
- Requires predicted structures (e.g., AlphaFold2) at inference, and accuracy drops for disordered regions with low pLDDT.
- Performance is limited on sequence-only datasets unless structures are predicted or the MST variant is used.
Experimental setup
Datasets: AlphaFoldDB (90% reduced, 18.8M structures) CATH43-S40 ProteinGYM DeepLoc Metal Ion Binding (TAPE) Thermostability (FLIP) DeepFRI GO annotations
Benchmarks: ProteinGYM DeepLoc Metal Ion Binding Thermostability GO annotations (MF, BP, CC)
Metrics: Spearman correlation NDCG Top-recall accuracy F1-max perplexity
Baselines: ESM-2 ESM-1v SaProt MIF-ST GearNet ESM-GearNet Tranception ESM-IF
Builds on: DeBERTa GVP SaProt Foldseek ESM-2
Closest papers by meaning
- Prot2Text-V2: Protein Function Prediction with Multimodal Contrastive Alignment (NeurIPS 2025 · similarity 0.89)
- Prot2Text: Multimodal Protein’s Function Generation with GNNs and Transformers (AAAI 2024 · similarity 0.89)
- ProtCLIP: Function-Informed Protein Multi-Modal Learning (AAAI 2025 · similarity 0.89)
- FoldToken: Learning Protein Language via Vector Quantization and Beyond (AAAI 2025 · similarity 0.88)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.88)
- ProtGO: Function-Guided Protein Modeling for Unified Representation Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.