Research map / Probabilistic and scientific modeling / Molecules and proteins

ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins

NeurIPS 2026 · Probabilistic and scientific modeling › Molecules and proteins · approach: Protein fitness and variant effect prediction · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ViroGym, a benchmark of 79 viral DMS assays, 21 influenza neutralisation tasks, and a SARS-CoV-2 GISAID pandemic-emergence task to evaluate zero-shot protein language models, finding ProGen2 strongest across all three.

Problem

Zero-shot protein language models have been validated mostly on non-viral proteins, leaving unclear how well they predict viral mutational fitness, antigenic change, and real-world emergent mutations needed for proactive vaccine design.

Approach

A benchmark that links in vitro viral DMS and neutralisation assays with real-world GISAID SARS-CoV-2 mutation frequencies, scoring off-the-shelf pLMs zero-shot and testing whether in vitro performance predicts real-world emergence.

Key results

• ProGen2 XL achieves the highest DMS top-10% recall (0.1973) and |Spearman| (0.2928), and the highest GISAID recall (0.4347) among 25 baselines.
• On neutralisation, Tranception M leads with Spearman 0.2316, while differences among models are marginal (about 0.19-0.23).
• Pseudo-temporal validation gives ProGen2 XL mean top-10% recall 0.39 and mean AUROC 0.74 across five SARS-CoV-2 variant eras; DMS and pLM top-10% RBD sets overlap only about 11% (Jaccard).

Surprising findings

• Predictions from ProGen2-XL and DMS assays barely overlap on top RBD mutations, with DMS recall for emergent mutations decaying from 30% in 2021 to 0% by 2025 KP.3.
• Semantic distance scoring of ESM2-650M outperforms likelihood-based scoring strategies on DMS, but ESM-family and ProtGPT2 models trail on pandemic prediction.

Limitations

• Pandemic prediction and real-world validation are limited to SARS-CoV-2 Spike; cross-virus extension infeasible due to GISAID data availability.
• DMS assay composition is skewed: SARS-CoV-2 is 47% and Influenza A 22% of assays.
• Model comparisons show marginal differences on neutralisation tasks, and the pandemic analysis uses a single reference-based frequency target rather than lineage forecasting.

Experimental setup

Datasets: 79 viral DMS assays (ProteinGym overlap plus newly curated) 21 influenza A neutralisation assays GISAID SARS-CoV-2 Spike mutation frequencies 2020-2025

Benchmarks: ViroGym DMS (79 assays) ViroGym influenza neutralisation (21 tasks) ViroGym SARS-CoV-2 GISAID pandemic prediction

Metrics: top-10% recall absolute Spearman correlation Precision@3 AUROC

Baselines: ESM1 ESM1v ESM2 ProtT5 ProtGPT2 Tranception VESPA VESPAl ProGen2

Builds on: ProteinGym EVEREST ESM-1v Tranception VESPA

Closest papers by meaning

• MutaPLM: Protein Language Modeling for Mutation Explanation and Engineering (NeurIPS 2024 · similarity 0.85)
• Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.85)
• DePLM: Denoising Protein Language Models for Property Optimization (NeurIPS 2024 · similarity 0.84)
• Ultrafast classical phylogenetic method beats large protein language models on variant effect prediction (NeurIPS 2024 · similarity 0.84)
• Augmenting Biological Fitness Prediction Benchmarks with Landscapes Features from GraphFLA (NeurIPS 2025 · similarity 0.83)
• Steering Protein Language Models (ICML 2025 · similarity 0.83)