Research map / Architectures and efficiency / Attention and transformers
MSA Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tied row attention (shared attention map across sequences) instead of untied row attention (ablation) | MSA Transformer · validation contact prediction, 100k updates | long-range P@L: more than 17 points | Helped |
| Use square-root normalization for tied row attention instead of mean normalization (ablation) | MSA Transformer tied row attention · validation contact prediction, 100k updates | long-range P@L: more than 6 points | Helped |
| Halve embedding size (384, 30M params) (ablation) | MSA Transformer · validation contact prediction, 100k updates | long-range P@L: 3.5 points lower than base model | Hurt |
| Select input sequences by greedy diversity maximization (or hhfilter) instead of random or diversity minimizing (ablation) | MSA Transformer inference · validation contact prediction, subsampled inputs | long-range P@L | Helped |
| Select input sequences by diversity minimizing (lowest hamming distance) instead of maximizing (ablation) | MSA Transformer inference · validation contact prediction, subsampled inputs | long-range P@L | Hurt |
| Shuffle MSA columns (destroy covariance) at inference (ablation) | MSA Transformer · MSAs with 1024 sequences, long-range P@L | long-range P@L: 52.9 to 15.9 | Hurt |
| Add MSA Transformer tied row attention and MSA input vs single sequence (ESM-1b-style) | unsupervised contact prediction · trRosetta test set | long-range P@L: +16.2 over ESM-1b (57.4 vs 41.1) | Helped |
| Use MSA Transformer features instead of Potts co-evolutionary features in supervised contact pipeline | supervised contact prediction · CASP13-FM | long-range P@L: 54.5 vs 40.1 (outer concat) | Helped |
| Use MSA Transformer representations for secondary structure prediction instead of ESM-1b or HMM profiles | CB513 secondary structure · CB513 8-class accuracy | accuracy: 73.4 vs 71.6 (ESM-1b) | Helped |
| Shuffle MSA positions (destroy sequence patterns) at inference (ablation) | MSA Transformer · MSAs with 1024 sequences, long-range P@L | long-range P@L: 52.9 to 27.9 | Hurt |
Closest papers by meaning
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.90)
- ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention (NeurIPS 2024 · similarity 0.90)
- MSA Generation with Seqs2Seqs Pretraining: Advancing Protein Structure Predictions (NeurIPS 2024 · similarity 0.89)
- From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models (ICML 2025 · similarity 0.89)
- SaProt: Protein Language Modeling with Structure-aware Vocabulary (ICLR 2024 · similarity 0.88)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.