Research map / Probabilistic and scientific modeling / Molecules and proteins
ProteinBench: A Holistic Evaluation of Protein Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ProteinBench, a holistic evaluation framework and leaderboard for protein foundation models spanning protein design (inverse folding, backbone, sequence, co-design, motif scaffolding, antibody design) and conformation prediction (folding, multi-state, distribution), scored on quality, novelty, diversity, and robustness.
Problem
Protein foundation models are evaluated with non-unified, task-specific metrics, making cross-model comparison and understanding of their strengths and limitations difficult.
Approach
A unified taxonomy of eight protein tasks with a shared multi-metric protocol (quality, novelty, diversity, and robustness/out-of-distribution checks) and user-objective-driven analyses, released as a living benchmark with leaderboard and modular toolkit.
Key results
- In de novo backbone design at length 500, RFdiffusion achieved the best scTM (0.79) and FrameFlow/Chroma remained competitive, with scTM dropping by over 20% for long chains across models.
- For sequence generation at length 100, DPLM (650M) achieved the best pLDDT (85.38) and ppl (6.21), while EvoDiff achieved the highest max cluster diversity (0.98).
- On CAMEO2022 folding, AlphaFold2 achieved mean TM-score 0.871 and lDDT 0.904, outperforming ESMFold (0.847) and EigenFold (0.743).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ProteinMPNN instead of language-model-based LM-Design for de novo backbone sequence design | Inverse-folding models · De novo backbones, length 100 | scTM: 0.834 -> 0.962 | Helped |
| Use LM--Design instead of ProteinMPNN to fit native evolutionary distributions | Inverse-folding models · CASP | AAR: 0.450 -> 0.516 | Helped |
| Use LM-Design instead of ProteinMPNN to fit native evolutionary distributions | Inverse-folding models · CAMEO | AAR: 0.468 -> 0.570 | Helped |
| Use DPLM for protein sequence generation rather than the other evaluated sequence generators | Sequence design models · Sequence design, length 100 | pLDDT: 85.38$\pm$14.20 | Helped |
| Use EvoDiff or ESM3 rather than DPLM when prioritizing sequence diversity | Sequence design models · Sequence design | diversity | Mixed |
| Use Multiflow rather than ProteinGenerator for co-design at longer lengths | Structure-sequence co-design models · Structure-sequence co-design, length 500 | scTM: 0.41$\pm$0.19 -> 0.83$\pm$0.15 | Helped |
| Add force guidance to ConfDiff | ConfDiff · BPTI, N=1000 | RMSDens (Å): 1.39$\pm$0.03 -> 1.36$\pm$0.01 | Helped |
| Increase sample depth N | Multiple-state prediction methods · BPTI | RMSDens (Å): 1.56$\pm$0.02 -> 1.46$\pm$0.00 | Helped |
| Use MSA subsampling to reduce MSA depth | MSA-based prediction · BPTI, Cluster 3 | RMSD (Å): 2.54$\pm$0.03 -> 1.85$\pm$0.05 | Helped |
| Fine-tune ESMFlow on MD conformation data rather than use the PDB-trained base model | ESMFlow · BPTI | diversity | Helped |
| Fine-tune on MD conformation data | AlphaFlow · ATLAS | Pairwise RMSD diversity | Helped |
| Use diversity-enhancing strategies for apo-holo state sampling | Multiple-state prediction models · Apo-holo | TMens | No clear effect |
Surprising findings
- On apo-holo multiple-state prediction, a baseline that always predicts the apo structure beats all sampling methods in TMens, suggesting current models struggle to capture apo-holo changes.
- No single inverse folding or antibody design method dominates across objectives; ProteinMPNN excels at de novo backbones but lags on native-distribution fitting.
Limitations
- Some methods (PiFold, CarbonDesign, Foldingdiff, Proteus) are not yet evaluated, and some results are quoted from prior papers or use released checkpoints trained on different data.
- Antibody design evaluation relies on predicted structures and computational proxies (Rosetta energies, IgFold) rather than experimental validation.
- FrameFlow results for long chains may be unfair because the released FoldFlow model was trained on shorter sequences.
Experimental setup
Datasets: CAMEO CASP15 RCSB PDB UniRef50 SAbDab RAbD CAMEO2022 apo-holo BPTI MD ATLAS
Benchmarks: CAMEO CASP15 CAMEO2022 apo-holo BPTI ATLAS RAbD motif scaffolding benchmark from RFdiffusion
Metrics: AAR scTM scRMSD pLDDT max TM to PDB pairwise TM max cluster perplexity binding energy TM-score
Baselines: ProteinMPNN ESM-IF1 LM-Design RFdiffusion FrameFlow Chroma ProGen2 EvoDiff DPLM ESM3 AlphaFold2 AlphaFlow
Builds on: AlphaFold2 ProteinMPNN RFdiffusion DiffAb AlphaFlow
Closest papers by meaning
- PSBench: a large-scale benchmark for estimating the accuracy of protein complex structural models (NeurIPS 2025 · similarity 0.91)
- Protein Structure Tokenization: Benchmarking and New Recipe (ICML 2025 · similarity 0.89)
- Proteina: Scaling Flow-based Protein Structure Generative Models (ICLR 2025 · similarity 0.88)
- CBGBench: Fill in the Blank of Protein-Molecule Complex Binding Graph (ICLR 2025 · similarity 0.88)
- CarbonNovo: Joint Design of Protein Structure and Sequence Using a Unified Energy-based Model (ICML 2024 · similarity 0.87)
- An All-Atom Generative Model for Designing Protein Complexes (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.