Research map / Probabilistic and scientific modeling / Molecules and proteins
Kernel-Based Evaluation of Conditional Biological Sequence Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes kernel-based ACMMD metrics and tests (accuracy and reliability) for conditional sequence models, and applies them to evaluate ProteinMPNN's fit on CATH proteins and temperature tuning.
Problem
Evaluating how accurately and how reliably conditional sequence models (e.g., inverse folding models for protein design) approximate the true conditional distribution of sequences given inputs, with absolute metrics that are estimable from samples and usable in hypothesis tests.
Approach
Defines the Augmented Conditional MMD (ACMMD) as the MMD between joint distributions P_X⊗P_{Y|X} and P_X⊗Q_{Y|X} with tensor-product kernels, gives an unbiased U-statistic estimator requiring only model samples, and builds a wild-bootstrap test with exact level control; extends to reliability (ACMMD-Rel) via a universal kernel on probability measures over sequences.
Key results
- ProteinMPNN at default temperature T=1.0 has ACMMD^2 of 0.0916 (p<0.01) on 5000 CATH proteins, rejecting that it fits the data.
- Lowering temperature below 1.0 improves fit and reliability, with optimal goodness-of-fit temperature at T=0.4 (outside the documented 0.1-0.3 range).
- ACMMD test controls type-I error at 0.05 on synthetic data and its power grows with sample size and shift magnitude.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reduce sampling temperature below 1.0 (ablation) | ProteinMPNN · CATH whole-data | goodness-of-fit (ACMMD²): T=1.0 | Helped |
| reduce sampling temperature below 1.0 (ablation) | ProteinMPNN · CATH whole-data | reliability (ACMMD--Rel²): T=1.0 | Helped |
| set sampling temperature to 0.4 (ablation) | ProteinMPNN · CATH whole-data | goodness-of-fit (ACMMD²): 0.4 | Helped |
| continue lowering sampling temperature (ablation) | ProteinMPNN · CATH whole-data | reliability (ACMMD--Rel²) | Helped |
| continue lowering sampling temperature (ablation) | ProteinMPNN · CATH whole-data | goodness-of-fit (ACMMD²) | Mixed |
| increase temperature difference from the reference model (ablation) | ProteinMPNN · CATH 1.10.10.10 | ACMMD²: δT ∈ {0, 0.01, 0.05, 0.1} | Helped |
| evaluate at different sampling temperatures (ablation) | ProteinMPNN · CATH superfamilies | goodness-of-fit (ACMMD²): T=1.0 | Mixed |
Surprising findings
- The goodness-of-fit optimal temperature (0.4) lies outside ProteinMPNN's suggested 0.1-0.3 range.
- Immunoglobulin superfamily has the best fit at low temperatures, yet this may not reflect CDR loop modeling.
Limitations
- Universality of kernels on sequence-valued spaces and on input structures is assumed or open; the structure/sequence kernels' universality is an open question.
- ACMMD-Rel estimator is biased (controlled by number of samples R) and only consistent in the limit.
- Metrics assess statistical fit on whole sequences and superfamilies; they do not establish experimental design success.
Experimental setup
Datasets: CATH S60 (17,540 single-domain monomer structures)
Benchmarks: CATH superfamilies synthetic toy sequence model
Metrics: ACMMD^2 estimate ACMMD-Rel^2 estimate test rejection rate p-value
Baselines: log-likelihood sequence recovery SKCE
Builds on: maximum mean discrepancy conditional mean embeddings squared kernel calibration error wild bootstrap
Closest papers by meaning
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.87)
- Kermut: Composite kernel regression for protein variant effects (NeurIPS 2024 · similarity 0.86)
- From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models (ICML 2025 · similarity 0.86)
- Contrastive losses as generalized models of global epistasis (NeurIPS 2024 · similarity 0.85)
- DePLM: Denoising Protein Language Models for Property Optimization (NeurIPS 2024 · similarity 0.85)
- Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Tasks (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.