Research map / Probabilistic and scientific modeling / Causal discovery
Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that synthetic experiments are necessary for evaluating causal ML, critiquing current practice, illustrating biases in RealCause and CausalNF failures, and proposing four evaluation principles.
Problem
Causal ML methods are hard to evaluate because counterfactual ground truth is unobservable and real causal datasets are scarce; current (semi-)synthetic evaluations are criticized as biased and too simple, hindering adoption.
Approach
Argues that rigorous, transparent synthetic experiments are essential rather than a flaw, and proposes principles (synthetic data necessity, explicit design choices across five components, evaluation beyond the identification domain with comprehensive metrics, standardized frameworks) plus illustrative experiments on RealCause and CausalNF.
Key results
- RealCause ATE error on IHDP varies widely: mean 0.38 ± 0.39 across 20 seeds for one realization, and 0.95 ± 1.36 across 100 realizations with a fixed seed, with some errors up to 9.45.
- CausalNF counterfactual RMSE is similar under diffeomorphism violations of Triangle NLIN (about 0.06–0.13), indicating robustness to those specific violations.
- On non-identifiable chain SCMs, CausalNF RMSE is 0.158 for CTF1 vs 0.583 for CTF2, suggesting it consistently converges to one structure regardless of truth.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| vary random seed instead of dataset realization in RealCause IHDP ATE evaluation | RealCause semi-synthetic IHDP benchmark · IHDP | ATE error: mean 0.38 ± 0.39 across seeds vs 0.95 ± 1.36 across realizations | Mixed |
| vary dataset realization across 100 IHDP realizations | RealCause · IHDP | ATE error: some realizations yield errors as high as 9.45 | Hurt |
| non-bijective noise transform (segmented linear, interval 0.2) added to Triangle_NLIN SCM (ablation) | CausalNF · Triangle_NLIN SCM | RMSE counterfactual: 0.132 vs 0.127 baseline | No clear effect |
| non-bijective sinusoidal noise transform added to Triangle_NLIN SCM (ablation) | CausalNF · Triangle_NLIN SCM | RMSE counterfactual: 0.061 vs 0.127 baseline at freq 0.25 (sinusoid variants range 0.060-0.085) | No clear effect |
| alternative counterfactual structure (CTF2) with same observational distribution as CTF1 (ablation) | CausalNF · non-identifiable chain SCM | RMSE counterfactual: 0.583 vs 0.158 for CTF1 | Hurt |
| noise-parent interaction in f2 and f3 (CTF4 vs CTF3) (ablation) | CausalNF · non-identifiable chain SCM | RMSE counterfactual: 1.037 vs 0.147 for CTF3 | Hurt |
Surprising findings
- Realism checks (KS tests) rarely identify subsets of RealCause runs with meaningfully lower ATE bias (only 1 of 100 realizations significant).
- CausalNF's counterfactual error does not increase under some non-bijective noise transformations, because the violation has little effect when noise does not interact with parents.
Limitations
- Experiments are illustrative rather than exhaustive; only RealCause and CausalNF are examined.
- Synthetic data cannot capture unknown unknowns, so it must be complemented with real-world experiments.
- Rigorous evaluation requires significant computational resources that may be unavailable to some researchers.
Experimental setup
Datasets: IHDP Triangle NLIN SCM (synthetic) chain SCMs CTF1-CTF4 (synthetic)
Benchmarks: IHDP Triangle NLIN SCM non-identifiable chain SCMs CTF1-CTF4
Metrics: ATE error RMSE for counterfactual estimation Welch's t-test p-values
Builds on: RealCause CausalNF IHDP benchmark
Closest papers by meaning
- Causality can systematically address the monsters under the bench(marks) (NeurIPS 2026 · similarity 0.88)
- Prediction-Powered Causal Inferences (NeurIPS 2025 · similarity 0.87)
- THE ROBUSTNESS OF DIFFERENTIABLE CAUSAL DISCOVERY IN MISSPECIFIED SCENARIOS (ICLR 2025 · similarity 0.87)
- Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.86)
- CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discovery (ICLR 2024 · similarity 0.86)
- Marginal Causal Flows for Validation and Inference (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.