Research map / Large language models / Agents, code and math
Automated Hypothesis Validation with Agentic Sequential Falsifications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Popper is an LLM-agent framework that validates free-form hypotheses by designing and executing sequential falsification experiments, aggregating p-values as e-values to control Type-I error.
Problem
Validating abstract, free-form natural language hypotheses (including LLM-generated ones) at scale while maintaining statistically rigorous Type-I error control.
Approach
Agents iteratively design falsification sub-hypotheses and execute tests on data, converting each p-value to an e-value via a p-to-e calibrator and multiplying them into an anytime-valid sequential e-process, with an LLM relevance checker ensuring each null is implied by the main hypothesis.
Key results
- Popper keeps Type-I error near nominal alpha=0.1 across DiscoveryBench (0.103), TargetVal-IL2 (0.082) and TargetVal-IFNG (0.085), while most baselines fail to control error.
- Among error-controlling methods, Popper gets 66.5% higher power than ReAct on DiscoveryBench and 3.17x Self-Refine power on TargetVal-IL2.
- Compared with nine computational biologists, Popper matched their error and power while completing tasks 9.7 times faster.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove the relevance checker (ablation) | Popper · DiscoveryBench | Type I error: 0.103\std{0.020} -> 0.134\std{0.021} | Hurt |
| remove the relevance checker (ablation) | Popper · TargetVal-IL2 | Type I error: 0.082 \std{0.046} -> 0.340\std{0.139} | Hurt |
| remove the relevance checker (ablation) | Popper · TargetVal-IFNG | Type I error: 0.085 \std{0.028} -> 0.300\std{0.113} | Hurt |
| replace ReAct with direct code generation (ablation) | Popper · DiscoveryBench | Power: 0.638*\std{0.066} -> 0.544\std{0.032} | Hurt |
| replace ReAct with direct code generation (ablation) | Popper · TargetVal-IL2 | Power: 0.580*\std{0.125} -> 0.526\std{0.133} | Hurt |
| replace ReAct with direct code generation (ablation) | Popper · TargetVal-IFNG | Power: 0.591*\std{0.069} -> 0.450\std{0.079} | Hurt |
| use sequential e-values instead of Fisher's combined test (ablation) | Popper · DiscoveryBench, TargetVal-IL2, and TargetVal-IFNG | Type I error: 0.311\std{0.040} -> 0.103\std{0.020}; 0.264\std{0.083} -> 0.082 \std{0.046}; 0.173\std{0.023} -> 0.085 \std{0.028} | Helped |
| use an LLM-estimated likelihood ratio instead of the p-to-e calibrator (ablation) | Popper · DiscoveryBench and TargetVal-IFNG | Type I error: 0.103\std{0.020} -> 0.152\std{0.031}; 0.085 \std{0.028} -> 0.180\std{0.028} | Hurt |
| increase the maximum number of tests (ablation) | Popper · benchmarks in sensitivity analysis | Power | Helped |
| use o1 as the LLM backbone instead of Claude-Sonnet-3.5 (ablation) | Popper · DiscoveryBench | Power: 0.638\std{0.066} -> 0.654*\std{0.019} | Mixed |
| use o1 as the LLM backbone instead of Claude-Sonnet-3.5 (ablation) | Popper · TargetVal-IL2 | Power: 0.580*\std{0.125} -> 0.336\std{0.121} | Hurt |
Surprising findings
- Removing the relevance checker substantially inflates Type-I error (e.g., TargetVal-IL2 rises to 0.340), and the LLM-likelihood-ratio baseline is miscalibrated in both directions.
- Claude-Haiku-3.5 has a Type-I error of 0.780 on TargetVal-IL2, showing weaker LLMs break error control.
Limitations
- Type-I error control does not guarantee discoveries are true, especially when many hypotheses are tested.
- Error analysis shows misinterpreted p-values (35.9%) and ineffective test design (28.1%) are the main failure modes.
- Evaluation is limited to a static database instantiation and small human study (9 participants).
Experimental setup
Datasets: DiscoveryBench TargetVal (22 tables, ~85M records from GTEx, GWAS Catalog, BioGrid) genome-wide CRISPR screen data
Benchmarks: DiscoveryBench TargetVal-IL2 TargetVal-IFNG
Metrics: Type-I error power task completion time
Baselines: CodeGen CodeGen (o1) ReAct Self-Refine Fisher Combined Test LLM-Likelihood ratio
Builds on: ReAct e-values e-processes Fisher's combined test Self-Refine
Closest papers by meaning
- Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing (NeurIPS 2026 · similarity 0.88)
- Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models (NeurIPS 2026 · similarity 0.85)
- Agentic-imodels: Evolving agentic interpretability tools via autoresearch (NeurIPS 2026 · similarity 0.85)
- ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence (NeurIPS 2026 · similarity 0.85)
- Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning (NeurIPS 2025 · similarity 0.85)
- MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.