Research map / Large language models / Agents, code and math

Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing

NeurIPS 2026 · Large language models › Agents, code and math · approach: Agent instruction-following benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces P-Bench, a 425-task expert-audited open-ended hypothesis-testing benchmark, and Fisher-R1, an open-weight R-coding agent trained with SFT on teacher trajectories then DAPO using a p-value/decision reward on synthetic tasks.

Problem

LLM agents performing hypothesis testing often choose invalid statistical methods and report incorrect p-values or conclusions even when code runs correctly; existing data-analysis benchmarks do not verify method validity or p-value accuracy.

Approach

Verified open-ended hypothesis-testing tasks with answer keys from canonical reference analyses, plus synthetic simulation-generated tasks whose answer keys are the output of the canonical method, enabling an outcome-grounded reward that scores p-value closeness in z-space and conclusion agreement with RL.

Key results

• Fisher-R1-14B reaches 33.0 Strict pass@1 on P-Hard vs 26.3 for DeepSeek V4 Pro and 30.5 for GPT-5.4.
• Fisher-R1-7B raises Raw pass@1 on P-Easy from 61.4 (Qwen2.5-Coder-7B) to 87.0 and Strict pass@1 from 36.3 to 65.7.
• Fisher-R1-14B reaches 64.2 Strict pass@1 on P-Easy vs 64.7 for GPT-5.4, nearly matching it.

Surprising findings

• GPT-5.4 gets 58.3 Raw but only 30.5 Strict on P-Hard, so conclusion-only evaluation overstates reliability.
• SFT alone did not improve pass@1 and DAPO alone plateaued well below SFT+DAPO.

Limitations

• P-Bench evaluates only a single hypothesis test per task, not multi-test pipelines with multiple-comparison correction.
• Method choice is evaluated only indirectly through p-value closeness; no explicit method-correctness reward.
• Reward uses an arbitrary z-space scale and sigma and the Strict threshold choice affects scores.

Experimental setup

Datasets: synthetic hypothesis-testing tasks (8,642) P-Bench (real economics, biology, medicine data)

Benchmarks: P-Bench

Metrics: Raw pass@1 Raw pass@3 Strict pass@1 Strict pass@3

Baselines: GPT-5.4 DeepSeek-V4-Pro GPT-OSS-120B Qwen3-Coder-30B Qwen3-32B DataMind-7B DataMind-14B Qwen2.5-Coder-7B Qwen2.5-Coder-14B

Builds on: DAPO ReAct GRPO CodeAct

Closest papers by meaning

• ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence (NeurIPS 2026 · similarity 0.89)
• The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.88)
• Automated Hypothesis Validation with Agentic Sequential Falsifications (ICML 2025 · similarity 0.88)
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.87)
• PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors (NeurIPS 2025 · similarity 0.87)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.87)