Research map / Probabilistic and scientific modeling / Causal discovery
Challenges and Considerations in the Evaluation of Bayesian Causal Discovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically examines whether common proxy metrics for Bayesian causal discovery (expected SHD, AUROC/AUPRC, NLL, interventional NLL/KL/MMD) track approximation quality against the true posterior, finding poor agreement in low-sample and non-identifiable (high-entropy) settings.
Problem
Evaluating Bayesian causal discovery is hard because the target is a posterior distribution, and there is no consensus on which proxy metrics reliably measure posterior approximation quality, especially with few samples or non-identifiable SCMs.
Approach
Uses linear Gaussian ANMs where the true posterior is tractable (d<6) to compare proxy metrics against MMD-based distances to the true graph and parameter posteriors, and links metric reliability to the entropy of the true posterior.
Key results
- With n=d=5 samples, the true posterior itself performs worse than several BCD approximations on common proxy metrics, and graph-based proxy metrics are not rank-correlated with Graph MMD.
- Proxy metrics become more correlated with true-posterior MMD at N=100 and especially for identifiable SCMs, but remain weakly correlated in non-identifiable low-data settings.
- True posterior entropy decreases with more samples and with identifiability, supporting the claim that current metrics are least reliable in high-entropy regimes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of observational training samples from 5 to 100 | Current proxy metrics for Bayesian causal discovery, d=5 · Synthetic linear ANMs | Rank correlation with true-posterior metrics | Helped |
| Use an identifiable rather than non-identifiable SCM | Current proxy metrics for Bayesian causal discovery · Synthetic linear ANMs, d=5 | Rank correlation with true-posterior metrics | Helped |
| Evaluate with current proxy metrics in a low-sample setting (n=5, d=5) | Bayesian causal discovery methods · Synthetic linear ANMs | Agreement in model rankings with true-posterior metrics | Hurt |
| Use graph-based proxy metrics to rank methods in low-sample settings | Bayesian causal discovery methods · Synthetic linear ANMs, n=5, d=5 | Rank correlation with Graph MMD | Hurt |
| Use Params MMD alongside the other proxy metrics in low-sample settings | Bayesian causal discovery methods · Synthetic linear ANMs, n=5, d=5 | Rank correlation with other metrics | Hurt |
| Increase observational data | True posterior · ER1 graphs, d=5 | Posterior entropy | Helped |
| Use an identifiable rather than non-identifiable model | True posterior · ER1 graphs, d=5 | Posterior entropy | Helped |
| Use DIBS in low-data regimes on SF graphs | DIBS · Scale-Free graphs | Posterior entropy | Hurt |
| Add interventional data and retrain at each step | VBG and BCD Nets · ER1 graphs | Model entropy | Helped |
| Add interventional data and retrain at each step | DIBS and BGIES · ER1 graphs | Model entropy | No clear effect |
| Use the true posterior as the reference instead of a single ground-truth graph | Posterior evaluation · Non-identifiable SCMs with limited data | Reliability of evaluation rankings | Helped |
| Evaluate downstream via causal effect estimation or Bayesian experimental design | Bayesian causal discovery evaluation · High-entropy true posterior settings | Evaluation suitability | Helped |
Surprising findings
- The true posterior itself scores worse than some approximate BCD methods on the common proxy metrics in low-sample regimes.
- DIBS produces very low-entropy posteriors and, in low-data SF graph settings, samples graphs with no edges because the prior dominates.
Limitations
- Restricted to causally sufficient linear additive noise models where the true posterior is tractable (d<6).
- Does not cover non-linear SCMs or settings with model misspecification.
- Only small graphs (d=5, 10) and a limited set of BCD methods are evaluated.
Experimental setup
Datasets: synthetic ER and SF graphs with linear Gaussian ANMs
Benchmarks: synthetic ER1/ER2/SF1/SF2 graphs with d=5
Metrics: E-SHD E-CPDAG SHD AUROC AUPRC NLL I-NLL I-KL I-MMD Graph MMD Params MMD
Baselines: BCD-Nets DIBS DIBS-Det DIBS-BGe VBG BGIES
Builds on: DIBS BCD-Nets VBG DAG bootstrap
Closest papers by meaning
- A Meta-Learning Approach to Bayesian Causal Discovery (ICLR 2025 · similarity 0.91)
- Bivariate Causal Discovery using Bayesian Model Selection (ICML 2024 · similarity 0.91)
- Continuous Bayesian Model Selection for Multivariate Causal Discovery (ICML 2025 · similarity 0.90)
- CausalTab: Pretraining Across Causal Environments for Tabular Causal Discovery (NeurIPS 2026 · similarity 0.89)
- THE ROBUSTNESS OF DIFFERENTIABLE CAUSAL DISCOVERY IN MISSPECIFIED SCENARIOS (ICLR 2025 · similarity 0.89)
- CausalRivers - Scaling up benchmarking of causal discovery for real-world time-series (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.