Research map / Probabilistic and scientific modeling / Molecules and proteins
Biological sequence transformers
Applies transformer language models to protein and genomic sequences.
34 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Molecules and proteins, keyword cluster “proteins · binding · sequence”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
10 papers in 2023–24 and 17 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 2.0% to 1.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Plan, Attend, Generate: Planning for Sequence-to-Sequence Models (NeurIPS 2017 · 12 citations)
- 2019 · Levenshtein Transformer (NeurIPS 2019 · 182 citations)
- 2021 · Transformer protein language models are unsupervised structure learners (ICLR 2021 · 372 citations)
- 2022 · Learning inverse folding from millions of predicted structures (ICML 2022 · 463 citations)
- 2023 · PiFold: Toward effective and efficient protein inverse folding (ICLR 2023 · 53 citations)
- 2024 · SaProt: Protein Language Modeling with Structure-aware Vocabulary (ICLR 2024 · 252 citations)
- 2025 · FoldToken: Learning Protein Language via Vector Quantization and Beyond (AAAI 2025 · 8 citations)
- 2026 · NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable Representations (AAAI 2026 · 2 citations)
Earliest papers here
- Plan, Attend, Generate: Planning for Sequence-to-Sequence Models (NeurIPS 2017 · 12 citations)
- Levenshtein Transformer (NeurIPS 2019 · 182 citations)
- Transformer protein language models are unsupervised structure learners (ICLR 2021 · 372 citations)
What has been tried
22 papers here report 248 measured changes (a component added, removed or swapped, compared with a baseline): helped 77%, no clear effect 5%, hurt 14%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Predict the residue token while revealing the structure token, rather than predicting both tokens | ProteinGym (w/o MSA retrieval) · Spearman’s ρ | 0.474 -> 0.478 | Helped | SaProt: Protein Language Modeling with Structure-aware Vocabulary (ICLR 2024 · 252 citations) |
| Replace K-mer tokenization with BPE | GUE (28 datasets) · average score | 60.92 -> 65.33 | Helped | DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes (ICLR 2024 · 173 citations) |
| Use pre-trained weights rather than training from scratch | Lung2 spatial transcriptomic imputation · Corr | 0.058 ± 0.020 -> 0.318 ± 0.015 | Helped | CellPLM: Pre-training of Cell Language Model Beyond Single Cells (ICLR 2024 · 75 citations) |
| use 2 MLP layers, code dimension 16, and spherical normalization rather than no normalization | CATH4.3 test set · TMScore | 0.5853 -> 0.6451 | Helped | FoldToken: Learning Protein Language via Vector Quantization and Beyond (AAAI 2025 · 8 citations) |
| add 8 histology prototypes rather than using 16 | TCGA six cohorts average · C-Index | 0.665 -> 0.655 | Hurt | Multimodal Prototyping for cancer survival prediction (ICML 2024 · 6 citations) |
| Add the Edge module | Protein-DNA, Protein-RNA, Protein-Protein · F1 | Helped | Protein-Nucleic Acid Complex Modeling with Frame Averaging Transformer (NeurIPS 2024 · 3 citations) | |
| Remove edge features | CATH 4.2, 15 layers · sequence recovery (%) | 54.08 -> 54.7 | Helped | De novo Protein Design Using Geometric Vector Field Networks (ICLR 2024 · 2 citations) |
| Add hierarchical residual quantization to VQDNA | Covid Variants Classification · F1 (%) | 73.16 -> 74.32 | Helped | VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling (ICML 2024 · 1 citation) |
| Add fixed-ratio curriculum learning with mask ratio 0.7 | Ablation study · AA Precision | 0.788 -> 0.733 | Hurt | Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing (ICML 2025 · 0 citations) |
| add one causal attention layer | PPBench complex design · design quality | Helped | PPDiff: Diffusing in Hybrid Sequence-Structure Space for Protein-Protein Complex Design (ICML 2025 · 0 citations) | |
| Increase static tree width substantially | Temperature 1, tree B versus tree C · tokens/sec. | 51.19 -> 46.90 | Hurt | STree: Speculative Tree Decoding for Hybrid State Space Models (NeurIPS 2025 · 0 citations) |
| Replace rule-based tokenization with learnt tokenization | Genomic Benchmarks · average score | 88.56 -> 88.89 | Helped | Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA (NeurIPS 2024) |
What reviewers will expect
Datasets and baselines named in the 8 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- SAbDab · 2 of 8 papers
- MT-Bench · 2 of 8 papers
- GSM8K · 2 of 8 papers
Baselines
- autoregressive decoding · 2 of 8 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 27 papers; from OpenAlex and the papers' affiliations).
- Westlake University · 4 papers
- Zhejiang University · 4 papers
- Zhejiang University & Westlake University · 2 papers
- Fudan University · 2 papers
- Carnegie Mellon University · 2 papers
- Yale University · 2 papers
Use it
Download BibTeX of 29 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Generative protein structure design (142 papers)
- Protein fitness and variant effect prediction (71 papers)
- Protein interface and docking prediction (72 papers)
- Protein representation learning (40 papers)
Elsewhere in the topic
- Prediction-powered and conformal inference (129 papers)
- Statistical analysis of ML practice (122 papers)
- Neuro-symbolic probabilistic layers (120 papers)
- Structure-based 3D molecule generation (107 papers)
- Sparse graphical and tensor models (97 papers)
- Flow matching generative modeling (97 papers)
- Machine learning for physical simulation (96 papers)
- Multimodal trajectory prediction (95 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).