Research map / Large language models / Agents, code and math
CellVerse: Do Large Language Models Really Understand Cell Biology?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CellVerse, a multiple-choice QA benchmark that converts single-cell multi-omics data into text (cell2sentence and GRN-based statements) and evaluates 14 open and closed LLMs on cell type annotation, drug response prediction, and perturbation analysis.
Problem
Whether LLMs can reason about cell biology when single-cell data are expressed as natural language QA, which has not been systematically evaluated.
Approach
Reformulate single-cell analysis at cell, drug, and gene levels as multiple-choice QA by serializing expression profiles as cell sentences and gene-regulatory relations as text, then benchmark a wide range of generalist and specialist LLMs under zero- and few-shot prompting.
Key results
- DeepSeek-R1 reaches 42.38% accuracy on scRNA-seq cell type annotation, and GPT-4.1 reaches 61.43% on CITE-seq and 29.33% on ASAP-seq (GPT-4.1-mini).
- On erlotinib drug response prediction, the best model (GPT-4.1-mini) reaches 55% accuracy, which the authors describe as not significantly better than random guessing.
- C2S-Pythia specialist models fail to give reasonable predictions across sub-tasks, while generalist DeepSeek and GPT-4 family models perform better.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use generalist LLMs instead of specialist C2S-Pythia models | CellVerse evaluation · Cell type annotation, drug response prediction, and perturbation analysis | prediction performance | Helped |
| Increase model capacity within evaluated LLM families | LLMs evaluated on CellVerse · CellVerse tasks | prediction performance | Helped |
| Increase context length by including more gene names | GPT-4 family · Cell type annotation, CITE-seq | accuracy | Helped |
| Increase context length by including more gene names | DeepSeek family · Cell type annotation, CITE-seq | accuracy | Mixed |
| Add few-shot examples | LLMs evaluated on CellVerse · Single-cell analysis tasks | performance | Mixed |
| Use LLMs for drug response prediction rather than random guessing | LLMs evaluated on CellVerse · Drug response prediction, scRNA-seq | accuracy: 55% | No clear effect |
| Use LLMs to predict perturbation direction rather than assess perturbation significance | LLMs evaluated on CellVerse · Perturbation analysis, scATAC-seq | accuracy: 76.67% vs 62.96% | Hurt |
Surprising findings
- Specialist C2S-Pythia models trained on single-cell tasks underperform generalist LLMs across all sub-tasks.
- Few-shot in-context learning does not consistently improve and sometimes degrades performance on single-cell tasks.
Limitations
- No quantitative difficulty levels assigned to tasks.
- All prompts and questions are English only.
- Open-ended questions were converted to multiple-choice, so results do not reflect free-form generation.
Experimental setup
Datasets: scRNA-seq multiple sclerosis data CITE-seq PBMC data ASAP-seq PBMC data erlotinib scRNA-seq drug response data K562 scATAC-seq perturbation data
Benchmarks: CellVerse
Metrics: precision recall F1 accuracy
Baselines: C2S-Pythia-160M C2S-Pythia-410M C2S-Pythia-1B Qwen-2.5-7B Qwen-2.5-32B Qwen-2.5-72B Llama-3.3-70B DeepSeek-V3 DeepSeek-R1 GPT-4 GPT-4o-mini GPT-4o
Builds on: cell2sentence gene regulatory network
Closest papers by meaning
- Cell2Sentence: Teaching Large Language Models the Language of Biology (ICML 2024 · similarity 0.92)
- CellPLM: Pre-training of Cell Language Model Beyond Single Cells (ICLR 2024 · similarity 0.90)
- scPilot: Large Language Model Reasoning Toward Automated Single-Cell Analysis and Discovery (NeurIPS 2025 · similarity 0.90)
- sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models (ICML 2025 · similarity 0.88)
- Cell ontology guided transcriptome foundation model (NeurIPS 2024 · similarity 0.87)
- LangCell: Language-Cell Pre-training for Cell Identity Understanding (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.