Research map / Large language models / Agents, code and math
Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SciGym, a dry-lab benchmark where LLM agents discover missing reactions of SBML systems by requesting simulated perturbation experiments and analyzing data, evaluating six frontier LLMs on 137 small systems.
Problem
Evaluating LLMs' iterative scientific competencies (experiment design and data interpretation) is infeasible with wet-lab experiments, which are expensive in expertise, time and equipment.
Approach
Use peer-reviewed SBML biological models from BioModels as a simulated "dry lab": the agent sees a model with reactions removed, requests initial-concentration perturbations, receives simulated time-series, and must recover the missing reactions, scored by reaction matching, topology F1, and trajectory error.
Key results
- Gemini-2.5-Pro achieved the best average STE of 0.3212 and RMS F1 of 0.3383 (without modifiers) among six models.
- Simulation error rose from about 0.1 to 0.55 as reactions increased from 2 to 10 for Claude-Sonnet and GPT-4.1.
- All models performed far worse on modifier relationships than reactant-product relationships (e.g., modifier precision near 0 for several models).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the SciGym agent framework with experiments and tools instead of zero-shot prompting | Six evaluated LLMs · SciGym-small | STE | Helped |
| Use the SciGym agent framework with experiments and tools instead of zero-shot prompting | Six evaluated LLMs · SciGym-small | RMS | Helped |
| Use the pro variant instead of the mini variant | Gemini-2.5 · SciGym-small | STE: 0.4181 -> 0.3212 | Helped |
| Use the pro variant instead of the mini variant | Gemini-2.5 · SciGym-small | RMS F1 (with modifiers): 0.1217 -> 0.1817 | Helped |
| Use the pro variant instead of the mini variant | GPT-4.1 · SciGym-small | STE: 0.6007 -> 0.4611 | Helped |
| Use the pro variant instead of the mini variant | GPT-4.1 · SciGym-small | RMS F1 (with modifiers): 0.1320 -> 0.1740 | Helped |
| Use the pro variant instead of the mini variant | Claude · SciGym-small | STE: 0.6281 -> 0.3615 | Helped |
| Use the pro variant instead of the mini variant | Claude · SciGym-small | RMS F1 (with modifiers): 0.0530 -> 0.1688 | Helped |
| Increase the number of reactions in the system | SciGym agents · SciGym-small biological systems | STE: close to 0.1 to 0.55 | Hurt |
| Identify modifier relationships rather than reactant-product relationships | All evaluated models · SciGym-small reaction networks | NTS | Hurt |
| Add Gaussian noise to initial concentrations | Agents' proposed SBML models · SciGym-small, perturbed initial conditions | STE | Hurt |
| Use species knockout perturbations instead of changing initial concentrations (ablation) | SciGym agents · SciGym-small | performance | Mixed |
Surprising findings
- Models almost entirely fail to recover reactant-modifier and modifier-product relationships, with precision often at or near 0.
- Agent-proposed models overfit observed data, with trajectory error increasing under perturbed initial conditions.
Limitations
- Simulation-to-reality gap: ODE-based SBML models simplify real biology.
- No experimental noise or measurement sparsity studied.
- Filtering removes models with rules and events, restricting coverage of biological systems and perturbations.
Experimental setup
Datasets: BioModels (350 filtered SBML models)
Benchmarks: SciGym-small
Metrics: Simulation Trajectory Error (SMAPE) Reaction Matching Score precision/recall/F1 Network Topology Score F1
Baselines: zero-shot prompting
Builds on: ReAct BioModels Tellurium
Closest papers by meaning
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.88)
- MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search (NeurIPS 2025 · similarity 0.88)
- ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows (ICLR 2026 · similarity 0.88)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.88)
- Language Modeling by Language Models (NeurIPS 2025 · similarity 0.87)
- PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.