Research map / Architectures and efficiency / RNNs and normalization
A Simple and Comprehensive Benchmark for Single-Cell Transcriptomics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks single-cell transcriptomics pre-training by addressing long-tailed values via weighted sampling, adapting MLP/CNN/Mamba alternatives to Transformers, and standardizing frozen-backbone downstream evaluations on expression and perturbation tasks.
Problem
Large-scale transformer pre-training on single-cell transcriptomics overlooks the long-tailed expression value distribution, questionable suitability of Transformer architectures for unordered gene data, and non-standardized, biology-specific evaluations.
Approach
Applies weighted sampling to counter the long-tailed value distribution, adapts MLP and CNN (with gene permutation) architectures to gene-expression data as alternatives to Transformers, and defines frozen-backbone, quantifiable downstream tasks (linear/KNN classification, perturbation classification and regression) as a fair benchmark.
Key results
- Weighted sampling (LTtrain) with MVCcls and log preprocessing raises average linear-probe accuracy across ten expression datasets from 79.259 to 83.323 relative to the random-sampling counterpart, and the paper reports 8.1% (linear) and 10.4% (KNN) average gains over the reproduced baseline.
- Adapted MLP and CNN achieve average linear accuracy of 81.4% and 81.2%, within 1.3% of the optimal 6-layer, 256-dim Transformer (84.956%) and above Mamba (81.677%) with fewer FLOPs.
- Transformer with optimal settings achieves the best perturbation classification on Adamson (45.841%) and regression losses on all three perturbation datasets, while Blood-only pre-training reaches 80.877 average accuracy and improves SLE classification.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add weighted sampling during training (ablation) | scGPT pre-training · Expression classification (10-dataset average) | accuracy (%): 79.218 -> 79.777 | Helped |
| add weighted sampling during testing (ablation) | scGPT evaluation · Expression classification (10-dataset average) | accuracy (%): 82.550 -> 83.323 | Helped |
| replace binning with log preprocessing (ablation) | scGPT · Expression classification (10-dataset average) | accuracy (%) | Helped |
| use MVC loss during training (ablation) | scGPT · Expression classification | accuracy (%) | Helped |
| use cls features instead of averaged token features for cell embeddings (ablation) | scGPT · Expression classification | accuracy (%) | Helped |
| adapt MLP to transform the gene dimension for feature interactions (ablation) | MLP · Expression classification (10-dataset average) | accuracy (%): 56.251 -> 81.415 | Helped |
| permute genes into a predefined order for CNN (ablation) | CNN · Expression classification (10-dataset average) | accuracy (%): 80.713 -> 81.233 | Helped |
| increase Transformer hidden dimensions from 256 to 768 (ablation) | Transformer · Expression classification (10-dataset average) | accuracy (%): 82.550 -> 83.376 | Helped |
| use sequence length 768 and 50% masking (ablation) | Transformer · Expression classification (10-dataset average) | accuracy (%): 82.550 -> 83.073 | Helped |
| increase pre-training data from 20% subset to the full dataset (ablation) | Transformer · Expression classification (10-dataset average) | accuracy (%): 82.262 -> 81.112 | Hurt |
| use Transformer instead of adapted MLP or CNN | pre-trained models · Expression classification (10-dataset average) | accuracy (%): 81.415 -> 84.956; 81.233 -> 84.956 | Helped |
| use Transformer instead of Mamba | pre-trained models · Dixit perturbation regression | test loss: 0.130 -> 0.042 | Hurt |
Surprising findings
- Pre-training on more data (54.6M cells) did not yield better downstream performance than organ-specific or 20% subsets, challenging the scaling assumption.
- Larger Transformer configurations (e.g., 12 layers, 512 dim) did not outperform smaller ones, contrary to the usual assumption that scale helps.
Limitations
- Weighted sampling is a naive strategy; more advanced long-tail methods are not explored.
- Adapted MLP and CNN still lag behind the optimal Transformer on perturbation classification (e.g., Norman) and regression tasks.
Experimental setup
Datasets: CELLxGENE (54.6M human cells) Myeloid Multiple Sclerosis Pancreas Checkpoint Inhibitor Colitis Myasthenia Gravis Systemic Lupus Erythematosus Zheng68K (scFoundation) Leptomeningeal Immune Cells Severe Dengue Leptomeningeal Metastasis Adamson
Benchmarks: Myeloid Multiple Sclerosis Pancreas Checkpoint Inhibitor Colitis Myasthenia Gravis Systemic Lupus Erythematosus Zheng68K Leptomeningeal Immune Cells Severe Dengue Leptomeningeal Metastasis Adamson Dixit
Metrics: test accuracy test loss
Baselines: scGPT Mamba MLP CNN Transformer
Builds on: scGPT masked language modeling Mamba
Closest papers by meaning
- PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis (NeurIPS 2025 · similarity 0.89)
- A benchmark for prediction of transcriptomic responses to chemical perturbations across cell types (NeurIPS 2024 · similarity 0.87)
- CellPLM: Pre-training of Cell Language Model Beyond Single Cells (ICLR 2024 · similarity 0.87)
- PRESCRIBE: Predicting Single-Cell Responses with Bayesian Estimation (NeurIPS 2025 · similarity 0.87)
- CellVerse: Do Large Language Models Really Understand Cell Biology? (NeurIPS 2025 · similarity 0.87)
- On the Recoverability of Causal Relations from Bulk Gene Expression Data (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.