Research map / Probabilistic and scientific modeling / Molecules and proteins
CellPLM: Pre-training of Cell Language Model Beyond Single Cells
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CellPLM, a pre-trained transformer that treats cells as tokens and tissues as sentences, pre-trained on scRNA-seq and spatial transcriptomics with a Gaussian mixture latent prior, and evaluated on denoising, imputation, annotation and perturbation prediction.
Problem
Existing single-cell pre-trained models treat genes as tokens and model only within-cell gene dependencies, ignoring cell-cell relations, and suffer from limited and noisy single-cell data.
Approach
Model a "cell language model" by treating cells as tokens within a tissue, masking gene expressions and predicting them from other unmasked cells; jointly pre-train on scRNA-seq and spatial transcriptomics with 2D sinusoidal positional encodings and a Gaussian mixture variational latent prior with a batch-aware decoder.
Key results
- Zero-shot clustering on a held-out dataset reaches ARI 0.867 and NMI 0.823, above PCA (0.843/0.812) and scGPT (0.836/0.818).
- Inference on 48,082 cells takes 0.85 s versus 129.19 s for scGPT and 428.24 s for Geneformer on an A100.
- Fine-tuned CellPLM achieves the best scRNA-seq denoising RMSE on PBMC 5K (0.725) and Jurkat (0.391) and the best Lung2/Liver2 imputation correlation (0.318/0.328), though the paper reports it outperforming all baselines across tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CellPLM | CellPLM · PBMC 5K | RMSE: 0.775 ± 0.002 -> 0.725 ± 0.001 | Helped |
| Fine-tune CellPLM | CellPLM · PBMC 5K | MAE: 0.565 ± 0.001 -> 0.551 ± 0.001 | Helped |
| Fine-tune CellPLM | CellPLM · Jurkat | RMSE: 0.423 ± 0.001 -> 0.391 ± 0.001 | Helped |
| Fine-tune CellPLM | CellPLM · Jurkat | MAE: 0.351 ± 0.001 -> 0.320 ± 0.001 | Helped |
| Fine-tune CellPLM | CellPLM · Lung2 | Corr: 0.227 ± 0.011 -> 0.318 ± 0.015 | Helped |
| Fine-tune CellPLM | CellPLM · Lung2 | Cosine: 0.360 ± 0.014 -> 0.481 ± 0.011 | Helped |
| Fine-tune CellPLM | CellPLM · Liver2 | Corr: 0.253 ± 0.014 -> 0.328 ± 0.011 | Helped |
| Fine-tune CellPLM | CellPLM · Liver2 | Cosine: 0.399 ± 0.012 -> 0.481 ± 0.010 | Helped |
| Fine-tune CellPLM | CellPLM · MS | F1: 0.703 -> 0.766 ± 0.007 | Helped |
| Fine-tune CellPLM | CellPLM · hPancreas | F1: 0.739 ± 0.006 -> 0.749 ± 0.010 | Helped |
| Fine-tune CellPLM | CellPLM · hPancreas | Precision: 0.761 ± 0.004 -> 0.753 ± 0.010 | Hurt |
| Use pre-trained weights rather than training from scratch (ablation) | CellPLM · Lung2 spatial transcriptomic imputation | Corr: 0.058 ± 0.020 -> 0.318 ± 0.015 | Helped |
Surprising findings
- Training CellPLM from scratch on the SRT imputation datasets hardly converges, indicating pre-training is necessary for that task.
- On the Liver2 imputation task, zero-shot CellPLM has a higher cosine variance and is not clearly better than the baselines in cosine similarity.
Limitations
- Gene set restricted to 13,500 common genes after inner join, and SRT data covers only 1,000 genes.
- Zero-shot performance is not consistently strong; the paper reports weaker results on some metrics.
- Attention-based cell-cell communication analysis is only a preliminary qualitative case study.
Experimental setup
Datasets: HTCA scRNA-seq HCA scRNA-seq GEO scRNA-seq CosMx NSCLC SRT PBMC 5K Jurkat Lung2 Liver2 MS hPancreas Adamson Perturb-Seq Norman Perturb-Seq
Benchmarks: PBMC 5K denoising Jurkat denoising Lung2 imputation Liver2 imputation MS cell type annotation hPancreas cell type annotation Adamson perturbation Norman perturbation
Metrics: ARI NMI RMSE MAE Pearson correlation cosine similarity F1 precision
Baselines: scGPT Geneformer PCA DeepImpute scGNN 2.0 GraphSCI SAVER DCA MAGIC scImpute SpaGE stPlus
Builds on: scBERT scGPT Flowformer SpaFormer scVI
Closest papers by meaning
- CellVerse: Do Large Language Models Really Understand Cell Biology? (NeurIPS 2025 · similarity 0.90)
- LangCell: Language-Cell Pre-training for Cell Identity Understanding (ICML 2024 · similarity 0.89)
- Cell2Sentence: Teaching Large Language Models the Language of Biology (ICML 2024 · similarity 0.89)
- sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models (ICML 2025 · similarity 0.87)
- A Simple and Comprehensive Benchmark for Single-Cell Transcriptomics (AAAI 2025 · similarity 0.87)
- Towards Scalable Context-Aware Single-Cell Spatial Transcriptomics Prediction from Histology Images (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.