Research map / Probabilistic and scientific modeling / Molecules and proteins
PSBench: a large-scale benchmark for estimating the accuracy of protein complex structural models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PSBench, a benchmark of over one million CASP15/CASP16 protein complex structural models with 10 quality scores, plus baseline EMA methods and evaluation tools, and demonstrates it with the GATE graph transformer, which ranked among top EMA predictors in CASP16.
Problem
Estimating the accuracy (EMA) of predicted protein complex structural models is hard because there are few large, diverse, well-annotated labeled datasets and standardized benchmarks for training and evaluating ML EMA methods.
Approach
Builds labeled datasets from blind CASP15 and CASP16 complex predictions (in-house MULTICOM models and community models), annotates each model with 10 global, local and interface quality scores via an automated pipeline, and trains the graph transformer GATE on CASP15 data to test it blindly in CASP16.
Key results
- On CASP16 in-house TOP5 models, GATE-AFM achieved the highest Spearman correlation for TM-score (0.283) and the lowest ranking loss (0.102), outperforming AFM Confidence (0.143 Spearman).
- In the CASP16 EMA competition, MULTICOM_GATE ranked first by Pearson correlation (0.673) for TM-score and third by Spearman correlation (0.456), ranking loss (0.135), and AUROC (0.652).
- PSBench contains over one million structural models (1,009,050 in-house CASP16 models plus about 12,904 community models) for 79 targets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GATE-AFM instead of AFM Confidence | CASP16 in-house TOP5 models · CASP16 in-house TOP5 | TM-score Spearman correlation: 0.143* -> 0.283 | Helped |
| Use GATE-AFM instead of AFM Confidence | CASP16 in-house TOP5 models · CASP16 in-house TOP5 | DockQ_wave Spearman correlation: 0.114* -> 0.322 | Helped |
| Train GATE on CASP15 community data and evaluate blindly | CASP16 EMA competition · CASP16 EMA competition, 37 targets | TM-score Pearson correlation: 0.673; ranked first | Helped |
| Train GATE on CASP15 community data and evaluate blindly | CASP16 EMA competition · CASP16 EMA competition, 37 targets | TM-score Spearman correlation: 0.456; ranked third | Helped |
Surprising findings
- PSS achieved the highest Pearson correlation (0.394) for TM-score on CASP16 in-house models, above GATE-AFM's 0.372.
Limitations
- Only 79 complex targets, which may not capture the full breadth of protein complexes.
- GATE-AFM was used only on top-ranked models due to time constraints in CASP16, and two large community targets were excluded from GATE evaluation.
- Four CASP16 targets with identical sequences were excluded from the in-house evaluation.
Experimental setup
Datasets: CASP15_inhouse_dataset CASP15_community_dataset CASP16_inhouse_dataset CASP16_community_dataset
Benchmarks: CASP16 in-house TOP5 models (32 targets) CASP16 EMA competition (37 targets)
Metrics: Pearson's correlation Spearman's correlation ranking loss AUROC
Baselines: AFM Confidence PSS GCPNet-EMA VoroMQA-dark VoroIF-GNN-score DProQA
Builds on: GATE AlphaFold2-Multimer AlphaFold3 DProQA
Closest papers by meaning
- ProteinBench: A Holistic Evaluation of Protein Foundation Models (ICLR 2025 · similarity 0.91)
- CBGBench: Fill in the Blank of Protein-Molecule Complex Binding Graph (ICLR 2025 · similarity 0.86)
- PSC-CPI: Multi-Scale Protein Sequence-Structure Contrasting for Efficient and Generalizable Compound-Protein Interaction Prediction (AAAI 2024 · similarity 0.86)
- Protein Structure Tokenization: Benchmarking and New Recipe (ICML 2025 · similarity 0.86)
- Evaluating Representation Learning on the Protein Structure Universe (ICLR 2024 · similarity 0.86)
- An All-Atom Generative Model for Designing Protein Complexes (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.