Research map / Large language models / Agents, code and math
LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LLM-SRBench, a 239-problem benchmark (LSR-Transform and LSR-Synth across four scientific domains) for LLM-based scientific equation discovery that reduces memorization, and benchmarks LLM-SR, LaSR, SGA and direct prompting with three LLM backbones.
Problem
Existing scientific equation discovery benchmarks (SRBench Feynman/Strogatz, SRSD) use well-known equations that LLMs can memorize, so they inflate LLM-based discovery performance and do not measure genuine data-driven discovery.
Approach
Build two problem sets that avoid recitation: LSR-Transform rewrites Feynman equations by solving for different pivot variables (checked with SymPy), and LSR-Synth combines LLM-proposed known terms with novel synthetic terms and generates data via numerical solvers, with OOD test splits; symbolic accuracy is judged by GPT-4o equivalence checks.
Key results
- Best method (LLM-SR with GPT-4o-mini) reaches 31.5% symbolic accuracy on LSR-Transform and LaSR with GPT-4o-mini reaches 28.1% on material science in LSR-Synth; overall peak is about 31%.
- GPT-4o-mini equivalence judgments agree with two human evaluators on 94.6% of 130 sampled problems.
- Direct prompting without data (DataBlind) performs poorly, e.g. 0-7% symbolic accuracy, showing data-driven refinement is needed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLM-SR instead of direct prompting without data-driven refinement | LLSR-Transform equation discovery · LSR-Transform | symbolic accuracy: 7.21 -> 31.53 | Helped |
| Use LLM-SR instead of direct prompting without data-driven refinement | LSR-Synth equation discovery · LSR-Synth Material Science | symbolic accuracy: 0.0 -> 20.24 | Helped |
| Use LaSR instead of LLM-SR | equation discovery methods · LSR-Transform | Acc0.1: 39.64 -> 50.45 | Helped |
| Use LaSR instead of LLM-SR | equation discovery methods · LSR-Synth Material Science | symbolic accuracy: 20.24 -> 28.12 | Helped |
| Use LLM-SR instead of LaSR | equation discovery methods · LSR-Synth Material Science | Acc0.1: 72.04 -> 88.28 | Helped |
| Use GPT-4o-mini rather than GPT-3.5-turbo | LLM-SR on LSR-Transform · LSR-Transform | symbolic accuracy: 10.81 -> 31.53 | Helped |
| Evaluate on OOD data rather than in-domain data | equation discovery methods · LSR-Synth | NMSE | Hurt |
| Use LSR-Transform problems rather than original Feynman problems | LLM-based equation discovery methods · LSR-Transform vs. Feynman | symbolic accuracy | Hurt |
Surprising findings
- Simple LLM sampling shows sharp numeric error drops and much lower symbolic error on original Feynman problems than on LSR-Transform, suggesting memorization.
- LSR-Transform problems remain harder than Feynman problems even at equal expression-tree complexity levels.
Limitations
- Symbolic accuracy relies on GPT-4o as judge, validated on only 130 problems with 94.6% agreement.
- LSR-Transform filtered to match Feynman complexity and to exclude problems solvable by direct sampling, which limits scope.
- Benchmark evaluates only three LLM backbones and a fixed budget of 1k LLM calls per problem.
Experimental setup
Datasets: Feynman equations (SRBench) LSR-Transform LSR-Synth
Benchmarks: LLM-SRBench (LSR-Transform) LLM-SRBench (LSR-Synth chemistry, biology, physics, material science)
Metrics: symbolic accuracy Acc_0.1 NMSE OOD NMSE
Baselines: LLM-SR LaSR SGA Direct Prompting (DataBlind)
Builds on: LLM-SR LaSR SGA SRBench Feynman benchmark
Closest papers by meaning
- LLM-SR: Scientific Equation Discovery via Programming with Large Language Models (ICLR 2025 · similarity 0.95)
- SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models (NeurIPS 2024 · similarity 0.88)
- MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search (NeurIPS 2025 · similarity 0.88)
- CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning (ICLR 2025 · similarity 0.88)
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.88)
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.