Research map / Large language models / Agents, code and math
Contextualizing biological perturbation experiments through language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PerturbQA, a QA benchmark turning Perturb-seq perturbation prediction (differential expression, direction of change, gene set enrichment) into LLM-answerable tasks, and SUMMER, a training-free summarize-retrieve-answer Llama3 framework that matches or exceeds prior SOTA.
Problem
Perturbation response prediction for unseen perturbations and downstream biological analyses (differential expression, gene set enrichment) are poorly served by regression objectives and graph adjacency matrices that discard edge semantics; current methods perform near random on discrete outcomes.
Approach
Reformulate perturbation prediction as discrete QA over textually-rich knowledge graphs and experimental outcomes, and use an inference-time LLM pipeline that summarizes KG neighborhoods, retrieves graph-proximal seen perturbation outcomes, and reasons via guided chain-of-thought prompting, without finetuning.
Key results
- SUMMER (Llama3 8B) reaches macro AUROC 0.58-0.61 on differential expression across K562, RPE1, HepG2, Jurkat and K562-Set, versus 0.48-0.58 for GEARS, scGPT and GenePT-based baselines in most cases.
- On direction of change, SUMMER achieves 0.62-0.69 AUROC, the best in 4 of 5 datasets, while GEARS reaches 0.64 on K562 only.
- In a single-biologist evaluation of 25 gene-set cases, the LLM summary was equal or more informative than over-representation analysis in 92% of cases and matched the manual annotation in 72%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Summer rather than the strongest listed baseline | Differential expression prediction · K562, RPE1, HepG2, Jurkat, K562-Set | AUROC: K562: 0.58 -> 0.60; RPE1: 0.58 -> 0.58; HepG2: 0.55 -> 0.61; Jurkat: 0.55 -> 0.58; K562-Set: 0.64 -> 0.61 | Mixed |
| Use Summer rather than the strongest listed baseline | Direction-of-change prediction · K562, RPE1, HepG2, Jurkat, K562-Set | AUROC: K562: 0.64 -> 0.62; RPE1: 0.60 -> 0.64; HepG2: 0.64 -> 0.65; Jurkat: 0.59 -> 0.66; K562-Set: 0.59 -> 0.69 | Helped |
| Use Summer descriptions rather than Combined gene set over-representation, top 10 | K562 gene clusters · K562 gene clusters | ROUGE-1 recall: 0.27 -> 0.57 | Helped |
| Use Summer 70B descriptions rather than Combined gene set over-representation, top 10 | K562 perturbation clusters · K562 perturbation clusters | ROUGE-1 recall: 0.63 -> 0.59 | Hurt |
| Replace manual gene set annotation comparison with Summer summary | K562 gene set enrichment · Human evaluation | summary equal or better than classical enrichment: 92% | Helped |
| Remove chain-of-thought-style guided reasoning (ablation) | Summer differential expression and direction prediction · All five datasets | AUROC: null | Hurt |
| Remove retrieved experimental outcomes (ablation) | Summer differential expression and direction prediction · All five datasets | AUROC: null | Hurt |
| Remove LLM processing and average retrieved labels (ablation) | Summer differential expression prediction · K562, RPE1, HepG2, Jurkat, K562-Set | AUROC: 0.58, 0.58, 0.55, 0.55, 0.64 vs Summer 0.60, 0.58, 0.61, 0.58, 0.61 | Mixed |
| Remove LLM processing and average retrieved labels (ablation) | Summer direction-of-change prediction · All five datasets | AUROC: 0.50, 0.50, 0.50, 0.50, 0.50 vs Summer 0.62, 0.64, 0.65, 0.66, 0.69 | Helped |
Surprising findings
- LLM baselines without retrieval (No CoT, No retrieval) perform no better than random on differential expression, showing that retrieved experimental outcomes are essential.
- Retrieval-only mean label (no LLM) is a strong baseline, outperforming GEARS and scGPT on differential expression.
Limitations
- Only single-gene perturbations and five Perturb-seq datasets; no combinatorial perturbations.
- Human evaluation is limited to 25 gene-set cases with one domain specialist.
- Smaller 8B model sometimes fails instruction following and shows incorrect causal directionality reasoning.
Experimental setup
Datasets: K562 genome-wide Perturb-seq (CRISPRi) K562 essential Perturb-seq RPE1 Perturb-seq HepG2 Perturb-seq Jurkat Perturb-seq STRING Gene Ontology Reactome CORUM UniProt Ensembl BioPlex
Benchmarks: PerturbQA
Metrics: macro AUROC ROUGE-1 recall BERT Score precision BERT Score recall BERT Score F1
Baselines: Physical (STRINGDB) GAT GEARS scGPT GenePT Over-representation analysis (Gene Ontology, Reactome, CORUM)
Builds on: GEARS retrieval-augmented generation chain-of-thought Graph RAG
Closest papers by meaning
- PerturbReason: A Knowledge-Grounded Benchmark and Framework for Cell-State–Conditioned Mechanistic Reasoning of Perturbation Effects (NeurIPS 2026 · similarity 0.91)
- PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis (NeurIPS 2025 · similarity 0.89)
- Chem-PerturBridge: a harmonized compendium of small molecule perturbation transcriptomic effects (NeurIPS 2026 · similarity 0.88)
- BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments (ICLR 2025 · similarity 0.88)
- MapPFN: Learning Causal Perturbation Maps in Context (NeurIPS 2026 · similarity 0.87)
- CellVerse: Do Large Language Models Really Understand Cell Biology? (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.