Research map / Label-efficient and robust learning / Recognition and long-tail
scMBERT: A Pre-Trained Deep Learning Model for Single-Cell Multiomic Data Representation and Prediction (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes scMBERT, a BERT-style pre-trained transformer for single-cell multiomic (scRNA-seq + scATAC-seq) data with variable input, evaluated on cell type annotation and batch-effect correction against scBERT and scGPT.
Problem
Existing single-cell foundation models are pre-trained only on scRNA-seq, and scBERT uses a fixed gene set that cannot handle new genes or modalities, while scGPT's GPT design suits ordered rather than unordered genomic input.
Approach
Adapts BERT to single-cell multiomic data with variable-length input of (value, feature name, modality) triplets, pre-trained with masked value prediction plus a data modality prediction task that discourages overfitting to one modality.
Key results
- scMBERT (Multiome) reaches 75.56% accuracy, AvgBIO 0.6362 and AvgBATCH 0.9072 on cell type annotation, versus scBERT 75.32% / 0.5687 / 0.8696 and scGPT 76.68% / 0.5272 / 0.8039.
- scMBERT fine-tuned on multiome data outperforms the RNA-only variant (74.26%, AvgBIO 0.6298, AvgBATCH 0.9013).
- scMBERT is comparable to SOTA on cell type accuracy but outperforms them on batch-effect correction metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with scRNA-seq and scATAC-seq instead of scRNA-seq only (ablation) | scMBERT · NeurIPS BMMC cell type annotation | Accuracy: 74.26% -> 75.56% | Helped |
| fine-tune with scRNA-seq and scATAC-seq instead of scRNA-seq only (ablation) | scMBERT · NeurIPS BMMC cell type annotation | AvgBIO: 0.6298 -> 0.6362 | Helped |
| fine-tune with scRNA-seq and scATAC-seq instead of scRNA-seq only (ablation) | scMBERT · NeurIPS BMMC cell type annotation | AvgBATCH: 0.9013 -> 0.9072 | Helped |
| use scMBERT (Multiome) instead of the strongest baseline, scGPT (RNA) | cell type annotation models · NeurIPS BMMC cell type annotation | Accuracy: 76.68% -> 75.56% | Hurt |
| use scMBERT (Multiome) instead of the strongest baseline, scBERT (RNA) | cell type annotation models · NeurIPS BMMC cell type annotation | AvgBIO: 0.5687 -> 0.6362 | Helped |
| use scMBERT (Multiome) instead of the strongest baseline, scBERT (RNA) | cell type annotation models · NeurIPS BMMC cell type annotation | AvgBATCH: 0.8696 -> 0.9072 | Helped |
Surprising findings
- scGPT has the highest raw accuracy (76.68%), above scMBERT (Multiome) at 75.56%, so the advantage is only on the batch and biology metrics.
Limitations
- Pre-trained on a relatively small amount of multiomic data (ENCODE only).
- Evaluated on a single downstream task (cell type annotation) plus batch correction; other tasks left for future work.
- Accuracy is comparable to, not better than, scGPT.
Experimental setup
Datasets: ENCODE 4 single-cell multiomic datasets NeurIPS 2021 BMMC multiome dataset
Benchmarks: NeurIPS 2021 BMMC multiome cell type annotation (3 held-out donors)
Metrics: accuracy AvgBIO AvgBATCH
Baselines: scBERT scGPT
Builds on: scBERT scGPT BERT
Closest papers by meaning
- scPilot: Large Language Model Reasoning Toward Automated Single-Cell Analysis and Discovery (NeurIPS 2025 · similarity 0.85)
- scSSL-Bench: Benchmarking Self-Supervised Learning for Single-Cell Data (ICML 2025 · similarity 0.84)
- Semi-supervised Knowledge Transfer Across Multi-omic Single-cell Data (NeurIPS 2024 · similarity 0.84)
- CellPLM: Pre-training of Cell Language Model Beyond Single Cells (ICLR 2024 · similarity 0.84)
- scMRDR: A scalable and flexible framework for unpaired single-cell multi-omics data integration (NeurIPS 2025 · similarity 0.84)
- sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.