Research map / Probabilistic and scientific modeling / Molecules and proteins
PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PerturBench, a modular codebase, datasets, tasks, and rank-based metrics for benchmarking single-cell perturbation response models, finding no single architecture dominates and simple baselines scale well.
Problem
Comparing single-cell perturbation response prediction models is hard because studies use inconsistent datasets, tasks and metrics, and models can suffer mode collapse that fit metrics (RMSE, cosine) fail to reveal.
Approach
A unified benchmarking framework with covariate-transfer and combinatorial tasks on six public datasets, a per-perturbation rank metric that detects mode collapse, and simple baselines (Linear, Latent Additive, Decoder-Only) evaluated alongside reimplemented published models (CPA, SAMS-VAE, Biolord, GEARS) under uniform HPO.
Key results
- On the Srivatsan transfer task, CPA* (noAdv) achieved the best rank metric (0.09 cosine LogFC rank) while SAMS-VAE* (S) had the best cosine LogFC (0.53).
- On the Jiang dataset, the Decoder-Only baseline was best on average response (cosine LogFC 0.64) and SAMS-VAE* was best on MMD PCA (0.32), with none of the published models matching simple baselines on rank metrics.
- On the McFaline data-scaling study, no model achieved a rank metric below 0.4, and simpler Latent Additive model outperformed CPA* and SAMS-VAE* at larger training sizes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove CPA* adversarial component (ablation) | CPA* · Srivatsan transfer | Cosine LogFC rank: 0.15 ± 1 × 10^{-2} -> 0.09 ± 4 × 10^{-3} | Helped |
| remove CPA* adversarial component (ablation) | CPA* · Norman combinatorial | Cosine logFC: 0.76 ± 4 × 10^{-3} -> 0.77 ± 1 × 10^{-2} | Helped |
| remove CPA* adversarial component (ablation) | CPA* · OP3 transfer | Cosine LogFC: 0.39 ± 6 × 10^{-3} -> 0.32 ± 6 × 10^{-3} | Hurt |
| remove CPA* adversarial component (ablation) | CPA* · Frangieh transfer | Cosine LogFC rank: 0.30 ± 3 × 10^{-2} -> 0.20 ± 3 × 10^{-3} | Helped |
| remove SAMS-VAE* sparse mask and global latent variables (ablation) | SAMS-VAE* · Srivatsan transfer | Cosine LogFC: 0.44 ± 1 × 10^{-3} -> 0.53 ± 1 × 10^{-2} | Helped |
| remove SAMS-VAE* sparse mask and global latent variables (ablation) | SAMS-VAE* · Norman combinatorial | Cosine logFC: 0.45 ± 2 × 10^{-2} -> 0.78 ± 6 × 10^{-3} | Helped |
| remove SAMS-VAE* sparse mask and global latent variables (ablation) | SAMS-VAE* · Jiang transfer | MMD PCA: 0.32 ± 5 × 10^{-3} -> 1.6 ± 1 × 10^{0} | Hurt |
| remove SAMS-VAE* sparse mask and global latent variables (ablation) | SAMS-VAE* · Frangieh transfer | Cosine LogFC: 0.15 ± 2 × 10^{-2} -> 0.22 ± 3 × 10^{-3} | Helped |
| replace gene-expression inputs with scGPT cell embeddings | CPA* and Latent Additive · Srivatsan transfer | Cosine LogFC: CPA*: 0.38 ± 6 × 10^{-3} -> 0.39 ± 9 × 10^{-3}; LA: 0.45 ± 2 × 10^{-3} -> 0.50 ± 4 × 10^{-3} | Helped |
| increase training data | perturbation response models · McFaline data-scaling scenario | Cosine LogFC | Mixed |
| increase training data | CPA* and CPA* (noAdv) · McFaline data-scaling scenario | Cosine LogFC | Hurt |
| increase training data | SAMS-VAE* and SAMS-VAE* (S) · McFaline data-scaling scenario | Cosine LogFC rank | Hurt |
Surprising findings
- Decoder (Cov) with no perturbation information fits cosine LogFC and RMSE well yet performs no better than random on rank metrics, exposing mode collapse.
- Ablating the adversarial component of CPA and the sparsity mask of SAMS-VAE improved performance over the original published models.
Limitations
- No single model architecture clearly outperforms others across datasets and metrics.
- Reimplementations of published models are assessed as core components, not exact recreations of public implementations.
- Models perform poorly on DEG recall and on the larger datasets, and noise in CRISPR knockdown labels may limit conclusions.
Experimental setup
Datasets: Srivatsan (sciplex3) Frangieh Jiang McFaline Norman19 OP3
Benchmarks: Srivatsan (sciplex3) transfer Frangieh transfer Jiang transfer McFaline data scaling Norman19 combinatorial OP3 transfer
Metrics: cosine logFC cosine logFC rank MMD PCA DEG recall RMSE mean RMSE mean rank
Baselines: CPA CPA (noAdv) SAMS-VAE Biolord GEARS Linear Latent Additive Decoder Only
Builds on: CPA SAMS-VAE GEARS Biolord optuna
Closest papers by meaning
- A benchmark for prediction of transcriptomic responses to chemical perturbations across cell types (NeurIPS 2024 · similarity 0.90)
- PerturbReason: A Knowledge-Grounded Benchmark and Framework for Cell-State–Conditioned Mechanistic Reasoning of Perturbation Effects (NeurIPS 2026 · similarity 0.90)
- PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction (ICML 2025 · similarity 0.90)
- A Simple and Comprehensive Benchmark for Single-Cell Transcriptomics (AAAI 2025 · similarity 0.89)
- Contextualizing biological perturbation experiments through language (ICLR 2025 · similarity 0.89)
- Chem-PerturBridge: a harmonized compendium of small molecule perturbation transcriptomic effects (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.