Research map / Large language models / Agents, code and math
DiscoveryBench: Towards Data-Driven Discovery with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DiscoveryBench, a benchmark of 264 real data-driven discovery tasks from published papers plus 903 synthetic tasks, with a facet-based (context/variable/relationship) LLM-judged evaluation showing the best agent reaches only 25%.
Problem
Evaluating how well LLMs can automate data-driven discovery, i.e., searching and verifying hypotheses from provided datasets, including semantic reasoning and workflow design beyond statistical analysis.
Approach
Formalizes data-driven hypotheses as context-variables-relationship triples organized in hypothesis semantic trees, builds real tasks from replicated published workflows and synthetic tasks from LLM-generated semantic forests, and scores predictions with a GPT-4-based facet-wise hypothesis match score (HMS).
Key results
- Best system reaches only about 25% HMS on DiscoveryBench; ReAct and DataVoyager do not improve over the simple CodeGen agent.
- Reflexion with oracle feedback significantly improves over CodeGen baseline.
- Biology (0%) and engineering (7%) are hardest domains; DataVoyager with GPT-4p drops from 17.5% to 9.9% when domain knowledge hints are added for archaeology tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add oracle HMS feedback and reflection between trials | CodeGen agent (Reflexion Oracle) · DB-Real and DB-Synth | HMS | Helped |
| Use ReAct instead of the simple CodeGen agent | Discovery agents · DB-Real and DB-Synth | HMS | No clear effect |
| Use DataVoyager planning with a self-critic instead of the simple CodeGen agent | Discovery agents · DB-Real and DB-Synth | HMS | No clear effect |
| Add domain knowledge | DataVoyager (GPT-4p) · Archaeology-related tasks in DB-Real | HMS: 17.5% -> 9.9% | Hurt |
Surprising findings
- Domain knowledge hints reduced DataVoyager performance on archaeology tasks (17.5% without vs 9.9% with), contrary to expectation.
- Common reasoning frameworks (ReAct, DataVoyager) fail to beat the simple CodeGen agent.
Limitations
- Excludes forecasting, simulation, and many natural/physical science domains; mostly social science tasks.
- Does not handle massive datasets or multi-modal/geospatial data.
- Real-task replication is costly (up to 90 person-hours per dataset) limiting scale.
Experimental setup
Datasets: DB-Real (264 tasks from published papers, incl. 25 train / 239 test) DB-Synth (903 synthetic tasks)
Benchmarks: DiscoveryBench (DB-Real) DiscoveryBench (DB-Synth)
Metrics: hypothesis match score (HMS) context F1 variable F1 relationship accuracy
Baselines: CodeGen ReAct DataVoyager Reflexion (Oracle) NoDataGuess
Builds on: ReAct Reflexion self-refine
Closest papers by meaning
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.88)
- LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models (ICML 2025 · similarity 0.88)
- MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search (NeurIPS 2025 · similarity 0.88)
- ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation (AAAI 2026 · similarity 0.88)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.88)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.