Research map / Vision-language and multimodal / Human and embodied modeling
ESM All-Atom: Multi-Scale Protein Language Model for Unified Molecular Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ESM-AA, a 35M-parameter protein language model that is pre-trained on code-switch sequences mixing residues with unzipped atoms plus multi-scale position encoding, to unify protein and small-molecule modeling.
Problem
Protein language models operate only at residue scale, so they cannot represent atom-level information needed for joint protein-molecule tasks, and atom-scale modeling of whole proteins is too costly.
Approach
Randomly unzip a small fraction of residues into their atoms to form code-switch sequences mixing residue and atom tokens, pre-train with masked language modeling over both scales and pairwise distance recovery, and encode positions with RoPE for residues plus a Gaussian-kernel 3D distance bias for atoms.
Key results
- On enzyme-substrate affinity regression (KM), ESM-AA with ProSmith reaches MSE 0.607 and R^2 0.560, versus 0.642 / 0.536 for ESM-2+Uni-Mol ProSmith.
- On drug-target affinity regression (Davis), ESM-AA with ProSmith reaches MSE 0.196 and r^2_m 0.752, the best reported on most metrics.
- On protein secondary structure and contact prediction, ESM-AA performs similarly to ESM-2 35M (e.g., CB513 SS3 0.79 vs 0.80), indicating retained protein understanding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ESM-AA in place of separate ESM-2 and Uni-Mol encoders with ProSmith | ProSmith · ESAR | MSE: 0.642 -> 0.607 | Helped |
| Use ESM-AA in place of separate ESM-2 and Uni-Mol encoders with ProSmith | ProSmith · DTAR (Davis) | MSE: 0.219 -> 0.196 | Helped |
| Use ESM-AA for secondary structure prediction | ESM-2 35M · SS3 (CB513) | ACC: 0.80 -> 0.79 | Hurt |
| Use ESM-AA for contact prediction | ESM-2 35M · Contact prediction, short range P@L | P@L: 0.20 -> 0.21 | Helped |
| Use ESM-AA for molecular property prediction | Uni-Mol · BACE | AUC: 83.2 -> 83.5 | Helped |
| Use ESM-AA for molecular property regression | Uni-Mol · QM7 | MAE: 58.9 -> 60.9 | Hurt |
| Use ESM-AA for zero-shot virtual screening | DUD-E · DUD-E zero-shot | AUROC: 81.72 -> 80.02 | Hurt |
| Remove atom-scale position encoding (ablation) | ESM-AA · ESAR | MSE: 0.627 -> 0.639(+0.012) | Hurt |
| Remove residue-scale position encoding (ablation) | ESM-AA · ESAR | MSE: 0.627 -> 0.676(+0.049) | Hurt |
| Remove MLM loss (ablation) | ESM-AA · ESAR | MSE: 0.627 -> 0.642(+0.015) | Hurt |
| Remove PDR loss (ablation) | ESM-AA · ESAR | MSE: 0.627 -> 0.645(+0.018) | Hurt |
| Remove molecule pre-training data (ablation) | ESM-AA · ESAR | MSE: 0.627 -> 0.648(+0.021) | Hurt |
Surprising findings
- Removing the pairwise distance recovery loss hurts more than removing the masked atom prediction loss in the ablation.
- On the DUD-E zero-shot virtual screening benchmark, ESM-AA (AUROC 80.02) is only slightly below DrugCLIP (81.72), which was retrained by the authors.
Limitations
- Cannot directly process SMILES; requires RDKit conformer generation for molecules.
- Molecular benchmark results are only comparable to Uni-Mol and do not uniformly beat it.
- Protein-molecule evaluations use frozen encoders within the ProSmith framework and the paper itself notes some baseline settings are not fully fair.
Experimental setup
Datasets: AlphaFold DB molecule dataset of 19M molecules and 209M conformations (Uni-Mol) KM Davis ESP CB513 CASP12 TS115 ProteinNet QM7 QM8 QM9
Benchmarks: KM Davis ESP CB513 CASP12 TS115 ProteinNet QM7 QM8 QM9 HIV MUV
Metrics: MSE R^2 Pearson ACC MCC ROC-AUC CI r^2_m SS3 accuracy SS8 accuracy
Baselines: ProSmith XGBoost Uni-Mol ESM-2 TAPE ResNet DrugCLIP GearNet
Builds on: ESM-2 ProSmith Uni-Mol RoPE
Closest papers by meaning
- An All-Atom Generative Model for Designing Protein Complexes (ICML 2025 · similarity 0.89)
- SaProt: Protein Language Modeling with Structure-aware Vocabulary (ICLR 2024 · similarity 0.87)
- Elucidating the Design Space of Multimodal Protein Language Models (ICML 2025 · similarity 0.87)
- Structure Language Models for Protein Conformation Generation (ICLR 2025 · similarity 0.87)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.87)
- From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.