Research map / Probabilistic and scientific modeling / Molecules and proteins
A benchmark for prediction of transcriptomic responses to chemical perturbations across cell types
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Open Problems Perturbation Prediction (OP3) living benchmark and a new scRNA-seq dataset of 146 compounds in human PBMCs from three donors, plus results from the NeurIPS 2023 single-cell perturbation prediction competition.
Problem
Predicting cell-type-specific transcriptomic responses to small-molecule perturbations is hampered by a lack of well-designed evaluation datasets and independent benchmarks, especially using primary human cells with donor replicates.
Approach
A formalized tensor-completion-style task (compounds x cell types x genes) on differential-expression representations (signed -log10 p-values from limma-voom), evaluated with MRRMSE and control methods within the Open Problems living benchmarking infrastructure, with a new multi-donor PBMC perturbation dataset and competition winners re-run as baselines.
Key results
- The new dataset profiles 146 compounds in PBMCs from three donors, retaining 301,785 cells and 5,317 filtered genes.
- The NeurIPS 2023 competition drew 1,318 participants from 1,097 teams submitting 25,529 solutions.
- Cross-donor retrieval analysis found significance-based (signed -log10 p-value) representations outperformed effect size and raw counts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use effect-significance values instead of effect sizes or conditional counts | OP3 perturbation representation · Cross-donor retrieval across cell types | retrieval rank | Helped |
| Add inferred gene regulatory networks from ATAC and RNA data (ablation) | Competition prediction models · Unseen-context perturbation prediction | prediction performance | Hurt |
| Add molecular-interaction and pathway-derived correlation information (ablation) | Competition prediction models · Unseen-context perturbation prediction | prediction performance | No clear effect |
| Use target encoding and singular value decomposition for preprocessing (ablation) | Competition prediction models · Leaderboard Track | MRRMSE | Helped |
| Add pseudolabels for model training | Neural network retraining · OP3 living benchmark | MRRMSE | Helped |
Surprising findings
- The top-performing method (NN retraining with pseudolabels) predicts high p-values more often than occur in the dataset.
- Incorporating inferred gene regulatory networks or molecular interaction networks as extra modalities decreased performance for many competitor models.
Limitations
- Only three healthy donors and a single cell type system (PBMCs), limiting generalization across donors and cell types.
- Single performance metric (MRRMSE) in Kaggle setting risks Goodhart-style overfitting.
- Strict gene filtering and removal of compounds with unclear annotations due to likely toxicity.
Experimental setup
Datasets: OP3 PBMC perturbation dataset
Benchmarks: OP3 perturbation prediction task
Metrics: MRRMSE MAE cosine similarity
Baselines: positive control negative control NN retraining with pseudolabels ChemBERTa-based winning method
Builds on: limma-voom Open Problems benchmarking infrastructure
Closest papers by meaning
- Chem-PerturBridge: a harmonized compendium of small molecule perturbation transcriptomic effects (NeurIPS 2026 · similarity 0.91)
- PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis (NeurIPS 2025 · similarity 0.90)
- A Simple and Comprehensive Benchmark for Single-Cell Transcriptomics (AAAI 2025 · similarity 0.87)
- PRESCRIBE: Predicting Single-Cell Responses with Bayesian Estimation (NeurIPS 2025 · similarity 0.87)
- PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction (ICML 2025 · similarity 0.86)
- PerturbReason: A Knowledge-Grounded Benchmark and Framework for Cell-State–Conditioned Mechanistic Reasoning of Perturbation Effects (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.