Research map / Large language models / Instruction and fine-tuning
Cell2Sentence: Teaching Large Language Models the Language of Biology
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Cell2Sentence, which converts single-cell gene expression profiles into rank-ordered gene-name "cell sentences" so that GPT-2 models can be fine-tuned to generate cells, predict cell types, and convert back to expression.
Problem
Applying pretrained large language models to single-cell transcriptomics, which is normally handled by specialized neural networks that do not leverage language pretraining.
Approach
Represent each cell as a text sequence of gene names ordered by expression rank (a "cell sentence"), which is approximately reversible to expression via a log-linear rank-expression fit, so standard causal LMs can be fine-tuned on it with natural-language prompts and metadata.
Key results
- GPT-2 Medium (NL + C2S) reaches 74.26% accuracy on autoregressive cell type prediction vs 29.17% for GPT-2 Medium (C2S) trained without natural-language pretraining.
- Linear rank-to-expression model captures over 81% of expression variation (R2) on a 50K-cell immune dataset; generated cells from GPT-2 Medium (NL + C2S) reach R2 0.943 against averaged real cells.
- In KNN classification of generated cells, GPT-2 Medium (NL + C2S) reaches up to 54.67% accuracy vs 43.06% for scVI at K=5 in expression space.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add natural-language pretraining before Cell2Sentence training (ablation) | GPT-2 Small · Immune tissue, 17 cell types | R2: 0.873 -> 0.949 | Helped |
| Add natural-language pretraining before Cell2Sentence training (ablation) | GPT-2 Medium · Unseen immune tissue test data | Accuracy: 29.17 -> 74.26 | Helped |
| Increase model size from GPT-2 Small to GPT-2 Medium | NL + C2S · Unseen immune tissue test data | Accuracy: 69.95 -> 74.26 | Helped |
| Generate cells with Cell2Sentence instead of scVI | Cell generation · Immune tissue, generated cells, K=5 | KNN accuracy (Expr.): 43.06 -> 54.67 | Helped |
| Use LoRA instead of full fine-tuning (ablation) | GPT-2 fine-tuning · Gene generation | Unique genes %: uniqueness of genes in generated sentences as low as 70% | Hurt |
| Use FP32 instead of FP16 (ablation) | GPT-2 training · Cell sentence training | Training speed: 60% slowdown | Hurt |
| Add natural-language pretraining before Cell2Sentence training (ablation) | GPT-2 Small · 35 cell types seen during training | Valid genes %: 97.84 -> 99.60 | Helped |
Surprising findings
- Models trained without natural-language pretraining fail to learn meaningful text embeddings, performing far worse on cell type prediction than NL-pretrained models.
- LoRA yielded highly variable generation with gene uniqueness as low as 70%, so full fine-tuning was used instead.
Limitations
- Rank encoding loses quantitative expression magnitude information, so the model must infer expression strength from gene order alone.
- Generated outputs need post-processing for invalid or duplicated gene names, and long cell sentences are computationally expensive.
- Experiments are limited to a restricted set of cell types and a single immune tissue dataset.
Experimental setup
Datasets: human immune tissue single-cell dataset (Domínguez Conde et al.) GEO/HCA-derived scRNA-seq datasets for reconstruction analysis (127 datasets)
Benchmarks: immune tissue held-out test set
Metrics: Pearson R Spearman R R2 KNN accuracy accuracy F1 perplexity gene validity gene uniqueness MMD
Baselines: scVI GPT-2 Small (C2S) GPT-2 Medium (C2S)
Builds on: GPT-2 scVI Scanpy LoRA
Closest papers by meaning
- CellVerse: Do Large Language Models Really Understand Cell Biology? (NeurIPS 2025 · similarity 0.92)
- CellPLM: Pre-training of Cell Language Model Beyond Single Cells (ICLR 2024 · similarity 0.89)
- Cell ontology guided transcriptome foundation model (NeurIPS 2024 · similarity 0.87)
- LangCell: Language-Cell Pre-training for Cell Identity Understanding (ICML 2024 · similarity 0.87)
- scPilot: Large Language Model Reasoning Toward Automated Single-Cell Analysis and Discovery (NeurIPS 2025 · similarity 0.86)
- sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.