Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Kermut: Composite kernel regression for protein variant effects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Kermut, a Gaussian process with a composite structure-and-sequence kernel for supervised protein variant effect prediction, achieving state-of-the-art ProteinGym Spearman and analyzing its uncertainty calibration.
Problem
Supervised protein variant effect prediction lacks reliable per-instance uncertainty estimates, which matter for protein engineering and Bayesian optimization.
Approach
A Gaussian process whose kernel multiplies Hellinger site-similarity, log-probability mutation similarity and inter-residue distance kernels (from ProteinMPNN inverse-folding distributions and AlphaFold structures), summed over mutations and added to an ESM-2 embedding squared-exponential sequence kernel, with a zero-shot ESM-2 mean function.
Key results
- Kermut reaches average Spearman 0.662 on ProteinGym substitutions vs 0.613 for ProteinNPT across contiguous, modulo and random splits.
- Ablation shows removing the structure kernel drops average Spearman by 0.065, the largest drop among components.
- Confidence-interval calibration is good overall (e.g., ECE 0.051 for main formulation), but error-based calibration per instance is weaker; ProteinNPT with MC dropout is overconfident.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Kermut's composite kernel instead of ProteinNPT | ProteinGym substitution benchmark · ProteinGym substitution, Avg. | Spearman: 0.613 -> 0.662 | Helped |
| Use Kermut's composite kernel instead of ProteinNPT | ProteinGym substitution benchmark · ProteinGym substitution, Avg. | MSE: 0.683 -> 0.589 | Helped |
| Add the structure kernel (ablation) | Kermut · ProteinGym substitution, Avg. (174 assays) | Spearman: -0.065 | Helped |
| Add the sequence kernel (ablation) | Kermut · ProteinGym substitution, Avg. (174 assays) | Spearman: -0.045 | Helped |
| Include inter-residue distance in the structure kernel (ablation) | Kermut · ProteinGym substitution, Avg. (174 assays) | Spearman: -0.035 | Helped |
| Include mutation probability and site comparison kernels (ablation) | Kermut · ProteinGym substitution, Avg. (174 assays) | Spearman: -0.026 | Helped |
| Use the zero-shot mean function instead of a constant mean (ablation) | Kermut · ProteinGym substitution, Avg. (174 assays) | Spearman: -0.026 | Helped |
| Include the mutation probability kernel (ablation) | Kermut · ProteinGym substitution, Avg. (174 assays) | Spearman: -0.016 | Helped |
| Include the site comparison kernel (ablation) | Kermut · ProteinGym substitution, Avg. (174 assays) | Spearman: -0.006 | Helped |
| Use the product of the four base kernels instead of Kermut's main sum formulation (ablation) | Kermut · ProteinGym, 174 ablation datasets | ECE: 0.051 -> 0.060 | Mixed |
Surprising findings
- MC dropout on ProteinNPT yields overconfident uncertainties, whereas the GP is better calibrated out of the box.
- Kermut trains in roughly 10 minutes for four datasets versus up to 400 hours for ProteinNPT.
Limitations
- O(N^3) GP scaling with dataset size limits multi-mutant settings.
- Cannot handle insertions or deletions because the site-comparison kernel relies on a fixed structure.
- No explicit epistasis modeling beyond the sequence kernel, which may hinder extrapolation to higher-order mutants.
Experimental setup
Datasets: ProteinGym substitution benchmark (217 DMS assays) ProteinGym multi-mutant assays (51)
Benchmarks: ProteinGym substitution benchmark
Metrics: Spearman correlation MSE expected calibration error expected normalized calibration error coefficient of variation
Baselines: ProteinNPT Tranception embeddings MSA Transformer (MSAT) embeddings ESM-1v embeddings TranceptEVE + OHE DeepSequence + OHE one-hot encoding (OHE)
Builds on: ProteinMPNN ESM-2 ProteinGym GPyTorch ProteinNPT
Closest papers by meaning
- Kernel-Based Evaluation of Conditional Biological Sequence Models (ICML 2024 · similarity 0.86)
- DePLM: Denoising Protein Language Models for Property Optimization (NeurIPS 2024 · similarity 0.83)
- Enhancing Protein Mutation Effect Prediction through a Retrieval-Augmented Framework (NeurIPS 2024 · similarity 0.83)
- Ultrafast classical phylogenetic method beats large protein language models on variant effect prediction (NeurIPS 2024 · similarity 0.83)
- Evolution-Inspired Loss Functions for Protein Representation Learning (ICML 2024 · similarity 0.82)
- Robust Model-Based Optimization for Challenging Fitness Landscapes (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.