Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks zero-shot single-cell foundation model embeddings (scBERT, Geneformer, scGPT, scFoundation, UCE) with MLP probes against raw-expression MLP, mean, and GEARS baselines for Perturb-seq effect prediction under SPECTRA-based distribution shift.
Problem
Whether contextualized zero-shot single-cell foundation model embeddings improve transcriptomic perturbation effect prediction over simple baselines, especially under distribution shift, which prior benchmarks did not account for.
Approach
A standardized evaluation framework that trains MLP probes on frozen scFM embeddings of control and in silico gene-knockout cells, tests them across SPECTRA sparsification splits, and summarizes robustness with AUSPC/ΔAUSPC, E-distance, and a cross-split contextual alignment metric.
Key results
- On Norman single-gene (2,000 HVGs), GEARS had AUSPC 0.00815 versus MLP baseline 0.0448 and scFMs performing slightly worse than the MLP baseline.
- On the top 20 DEGs, UCE was the best scFM for Norman single-gene (AUSPC 0.334) and ranked second overall behind GEARS (0.266), with the MLP baseline at 0.342.
- On Replogle K562, GEARS achieved AUSPC 0.0082 while scGPT and UCE (0.1384) ranked above the MLP baseline (0.1420), with differences among non-GEARS models not statistically significant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GEARS instead of zero-shot scFM embeddings | Perturbation effect prediction · Norman single-gene (2,000 HVGs) | AUSPC: 4.525 ± 0.255 -> 0.815 ± 0.039 | Helped |
| Use GEARS instead of the MLP baseline | Perturbation effect prediction · Norman double-gene (2,000 HVGs) | AUSPC: 4.253 ± 0.073 -> 0.808 ± 0.028 | Helped |
| Use scBERT embeddings instead of the mean baseline | Perturbation effect prediction · Replogle K562 (2,000 HVGs) | AUSPC: 0.1386 ± 0.0076 -> 0.1228 ± 0.0016 | Helped |
| Use Geneformer instead of the mean baseline | Perturbation effect prediction · Replogle RPE1 (2,000 HVGs) | AUSPC: 0.1341 ± 0.0035 -> 0.1251 ± 0.0023 | Helped |
| Use zero-shot scFM embeddings instead of raw gene-expression features | Perturbation effect prediction · Norman single-gene (2,000 HVGs) | AUSPC | Mixed |
| Use zero-shot scFM embeddings instead of simple baselines | Perturbation effect prediction · Norman double-gene (top 20 DEGs) | AUSPC: 0.371 ± 0.009 -> 0.409 ± 0.008 | Hurt |
| Increase sparsification probability | All evaluated models · Norman single- and double-gene perturbations | MSE | Hurt |
| Increase probe parameter count (ablation) | MLP probes on raw gene expression and scFM embeddings · Perturbation effect prediction | prediction performance | No clear effect |
| Represent CRISPRa perturbations by expression doubling instead of nullifying the target gene (ablation) | scFM in-silico perturbation representation · Perturbation effect prediction | prediction performance | No clear effect |
Surprising findings
- scGPT's performance dropped sharply on Norman double-gene perturbations at high sparsification (MSE 14.906 at split 0.7), falling from rank 3 to rank 8.
- scFM embeddings offered limited improvement over simple mean and MLP baselines, and a task-specific GEARS model outperformed all zero-shot scFMs by an order of magnitude in AUSPC.
Limitations
- Perturbations are modeled only as knockouts (nullifying expression), so CRISPRa perturbations on Norman cannot be represented faithfully by all scFMs.
- Contextual alignment analysis covers only scGPT and scBERT on the Norman dataset.
- Compute is described as one A100 GPU per MLP probe, with no full compute accounting.
Experimental setup
Datasets: Norman et al. 2019 Perturb-seq K562 Replogle et al. 2022 K562 Replogle et al. 2022 RPE1
Benchmarks: Norman single-gene Norman double-gene Replogle K562 Replogle RPE1
Metrics: MSE AUSPC delta AUSPC E-distance contextual alignment (cross-split overlap)
Baselines: MLP baseline mean baseline GEARS
Builds on: SPECTRA GEARS Perturb-seq
Closest papers by meaning
- PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis (NeurIPS 2025 · similarity 0.90)
- A benchmark for prediction of transcriptomic responses to chemical perturbations across cell types (NeurIPS 2024 · similarity 0.86)
- Contextualizing biological perturbation experiments through language (ICLR 2025 · similarity 0.86)
- PRESCRIBE: Predicting Single-Cell Responses with Bayesian Estimation (NeurIPS 2025 · similarity 0.86)
- PerturbReason: A Knowledge-Grounded Benchmark and Framework for Cell-State–Conditioned Mechanistic Reasoning of Perturbation Effects (NeurIPS 2026 · similarity 0.85)
- MapPFN: Learning Causal Perturbation Maps in Context (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.