Research map / Probabilistic and scientific modeling / Causal discovery

Causality can systematically address the monsters under the bench(marks)

NeurIPS 2026 · Probabilistic and scientific modeling › Causal discovery · approach: Causal approaches to fairness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing that causal graphs and inference should guide evaluation of ML and LLM reasoning, introducing Common Abstract Topologies (CATs) and illustrating them through case studies.

Problem

Benchmarks and evaluations of large models suffer from biases, artifacts, leakage, and unexplored failure modes, and are often treated haphazardly with implicit, inconsistent assumptions, leading to unreliable conclusions about reasoning abilities.

Approach

Frame the recurring evaluation "monsters" (confounding, mediation, spurious correlations) as explicit causal structures, and offer three simple causal graph templates (CATs) to make causal modeling accessible for evaluating LLM reasoning.

Key results

• Presents three Common Abstract Topologies (confounding, mediation, spurious correlations) as causal templates for ML evaluation issues.
• Reanalyzes a term-frequency study on arithmetic reasoning through confounding and spurious-correlation graphs, proposing a uniform-sampling alternative benchmark.
• Argues that a label-bias mitigation via local token edits needs a strong independence assumption that a causal regularization view avoids.

Surprising findings

• Causality-related terms rarely co-occur with reasoning-related terms in ACL Anthology papers despite the rapid growth of reasoning research.

Limitations

• No empirical experiments; the case studies are illustrative reanalyses of prior work.
• Causal graphs are often underdetermined by data, so the framing may be contested.
• Focus is largely on NLP and LLM reasoning, with claims extended to empirical ML only by argument.

Experimental setup

Datasets: GSM8K

Builds on: common cause principle item response theory induction heads competency problems

Closest papers by meaning

• Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? (NeurIPS 2024 · similarity 0.89)
• Can Large Language Models Infer Causation from Correlation? (ICLR 2024 · similarity 0.89)
• Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption (ICML 2025 · similarity 0.88)
• Causal Order: The Key to Leveraging Imperfect Experts in Causal Inference (ICLR 2025 · similarity 0.88)
• Since Faithfulness Fails: The Performance Limits of Neural Causal Discovery (ICML 2025 · similarity 0.88)
• Causal Concept Graph Models: Beyond Causal Opacity in Deep Learning (ICLR 2025 · similarity 0.88)