Research map / Multi-agent systems and games / LLM agents

Can AI Agents Synthesize Scientific Conclusions?

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SciConBench, a live benchmark of 9.11K Cochrane review question–conclusion pairs, and SciConHarness, a clean-room web-agent harness that filters ground-truth leakage, to evaluate scientific conclusion synthesis by frontier models and deep research agents.

Problem

Scientific AI agents increasingly synthesize conclusions from open-web evidence in high-stakes health settings, but existing benchmarks are small, static, non-agentic, and vulnerable to leakage of ground-truth review conclusions.

Approach

A live, large-scale benchmark built from Cochrane systematic review objectives and authors' conclusions, paired with a clean-room MCP harness that filters out ground-truth reviews, and an atomic-fact precision/recall (F1) evaluation using LLM judges validated against experts.

Key results

• Best clean-room system o3-deep-research reaches factual F1 of 0.337 on SciConBench.
• Clean-room evaluation reduces factual F1 by 0.02 to 0.172 across systems relative to unconstrained tool use.
• Consumer-facing agents achieve F1 of 0.361-0.522 (OpenEvidence 0.522, Google AI Overview 0.384, Google AI Mode 0.361), with 50.8-59% of conclusions containing at least one contradicted fact.

Surprising findings

• Clean-room filtering consistently lowers performance, suggesting leakage inflates measured synthesis ability.
• Adding web tools degraded gemini-3-pro and gave only marginal gains for gpt-5.1 despite heavier tool use.

Limitations

• Restricted to science and health domains derived from Cochrane reviews; may not generalize to other fields.
• Residual indirect leakage (e.g., news coverage) may persist despite clean-room filtering.
• Evaluation of deep research agents on the full 268-sample set is cost-limited; some used subsets or capped tool calls.

Experimental setup

Datasets: Cochrane Database of Systematic Reviews (CDSR)

Benchmarks: SciConBench

Metrics: factual precision factual recall factual F1 Gwet's AC1 Cohen's kappa

Baselines: gpt-5.1 claude-sonnet-4.5 gemini-3-pro sonar-reasoning-pro sonar-deep-research o4-mini-deep-research o3-deep-research DR Tulu Google AI Overview Google AI Mode OpenEvidence

Builds on: DR Tulu dr-agent-lib FActScore Min et al. atomic fact decomposition

Closest papers by meaning

• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.89)
• ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence (NeurIPS 2026 · similarity 0.89)
• SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.88)
• InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.88)
• MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome (NeurIPS 2026 · similarity 0.87)
• From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents (NeurIPS 2026 · similarity 0.87)