Research map / Large language models / AI and society
Position: Data-driven Discovery with Large Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing for end-to-end data-driven discovery systems built on large generative models, illustrated by the GPT-4 multi-agent proof-of-concept DATAVOYAGER and its limitations.
Problem
Scientists struggle to absorb, search, and verify hypotheses from the growing volume of available data; prior discovery systems lacked compute, were domain-specific, or required human wet-lab work for verification.
Approach
Proposes that large generative models, combined with fail-proof tool integration and user feedback, can automate hypothesis search and statistical verification purely from provided datasets, and demonstrates this with DATAVOYAGER, a role-based multi-agent GPT-4 system built on AutoGen.
Key results
- DATAVOYAGER with GPT-4 qualitatively reproduces data understanding, hypothesis generation, multi-step planning, and statistical verification (e.g., t-tests, OLS, GLM) on National Longitudinal Surveys examples.
- Survey in Figure 3 finds existing systems (e.g., CoScientist, DataLume, AutoML tools) each cover only some desired functionalities of an ideal discovery system.
- The paper reports no quantitative benchmark results; evidence is qualitative case examples.
Surprising findings
- Authors note DATAVOYAGER often deviates when fully unsupervised, e.g., re-focusing on multicollinearity despite a different objective, requiring user intervention.
Limitations
- LGMs hallucinate and lack System-2 reasoning, needing scaffolding for long-horizon tasks.
- Scalability to large datasets and long-context analysis remains unsolved; no cost-at-scale controls.
- Risks of p-hacking, policy misuse, legal liability, and bias from datasets and LGMs.
Experimental setup
Datasets: National Longitudinal Surveys
Baselines: CoScientist DataLume MLAgentBench Bacon
Builds on: AutoGen Reflexion AutoML
Closest papers by meaning
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.87)
- Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges (AAAI 2025 · similarity 0.86)
- MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search (NeurIPS 2025 · similarity 0.85)
- LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models (ICML 2025 · similarity 0.85)
- Prediction-Augmented Trees for Reliable Statistical Inference (NeurIPS 2026 · similarity 0.85)
- FIND: A Framework for Discovering Formulas in Data (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.