Research map / Large language models / Agents, code and math

ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence

NeurIPS 2026 · Large language models › Agents, code and math · approach: Long-horizon LLM agents

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ACED-Bench, a neuro-symbolic benchmark where hidden probabilistic graphical models back LLM-written causal scenarios, testing whether LLM agents can acquire and act on causal evidence under a four-turn budget.

Problem

Existing benchmarks do not jointly test whether LLM agents can design experiments, interpret finite sampled evidence, and make causal decisions in a world that is both readable and formally consistent with exact ground truth.

Approach

Separate the readable scientific surface (LLM-written scenarios and task text) from a hidden PGM that defines the causal graph, CPDs, intervention semantics, validation checks and oracle answers, so gold labels are computed programmatically and agents must actively sample observational and interventional data.

Key results

• Strong active agents approach ceiling on ACED-Qual, while ACED-Core retains substantial spread across models and agent designs.
• Samp5k sample-budget ablation: Active+Code improves in 8 of 9 model cells with a mean gain of 7.3 accuracy points over the four-turn setting.
• Evidence-ledger analysis over 3,579 modular active-code trajectories shows failures from missed estimands, lost evidence, and continued querying after sufficient evidence.

Surprising findings

• Single-prompt active-code can underperform active reasoning under the four-turn budget because the code loop consumes scarce turns.
• Relaxing the constraint from turns to sampled rows (Samp5k) changes which agent design benefits most, with single-prompt active-code gaining most.

Limitations

• Covers only fully observed categorical PGMs; no continuous variables, latent confounding, temporal dynamics, or measurement error.
• Scenarios are synthetic and the benchmark instance is modest (180 ACED-Core and 150 ACED-Qual tasks).
• Evidence ledger is a diagnostic based on LLM annotation rather than a separate benchmark score.

Experimental setup

Benchmarks: ACED-Core ACED-Qual Samp5k

Metrics: accuracy exact match F1 tie-accepted accuracy evidence-ledger acquisition and retention

Baselines: GPT-4o Llama-3.3-70B guess-only agent

Builds on: Bayesian networks d-separation

Closest papers by meaning

• Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing (NeurIPS 2026 · similarity 0.89)
• Can AI Agents Synthesize Scientific Conclusions? (NeurIPS 2026 · similarity 0.89)
• Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? (NeurIPS 2024 · similarity 0.88)
• LLM Agents Can Be Choice-Supportive Biased Evaluators: An Empirical Study (AAAI 2025 · similarity 0.88)
• InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.88)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.88)