Research map / Probabilistic and scientific modeling / Causal discovery
Do-PFN: In-Context Learning for Causal Effect Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a TabPFN-style transformer (Do-PFN) on synthetic SCMs with simulated interventions so it predicts interventional outcomes and CATEs from observational data alone, without the causal graph.
Problem
Estimating causal effects (conditional interventional distributions and CATEs) from observational data typically requires interventional data, a known causal graph, or unconfoundedness, which limits real-world applicability.
Approach
Extend prior-data fitted networks to causal inference by pre-training on synthetic SCMs that simulate both observational data and interventional outcomes, so the model learns to output interventional distributions p(y|do(t), x) conditioned on an observational dataset in-context, with theoretical consistency and optimality results.
Key results
- Do-PFN significantly outperforms Random Forest, TabPFN (v2) and Dont-PFN in normalized MSE for CID prediction across synthetic case studies.
- Do-PFN significantly outperforms meta-learners, DML, DragonNet and TARNet on CATE estimation in the synthetic known-ground-truth case studies.
- On RealCause, Do-PFN (v1.1) is competitive with strong CATE baselines in unconfounded settings, and on Law School it achieves the strongest performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use intervention-focused pre-training instead of observational-outcome pre-training | Do-PFN vs Dont-PFN · Synthetic CID case studies | MSE | Helped |
| Use Do-PFN instead of Random Forest, TabPFN v2, or Dont-PFN | Synthetic CID prediction · Synthetic case studies | MSE | Helped |
| Use Do-PFN instead of meta-learners, DML, DragonNet, or TARNet | CATE estimation · Synthetic case studies | MSE | Helped |
| Use Do-PFN instead of causal baselines | CATE estimation · Law School Admissions | MSE | Helped |
| Use Do-PFN in place of other methods | CATE estimation · Amazon Sales | MSE | Mixed |
| Extend pre-training to graph sizes up to 60 nodes (ablation) | Do-PFN v1.1 · Synthetic graphs with 21–50 nodes | MSE | Helped |
| Increase the number of observational data points (ablation) | Do-PFN · Synthetic CATE datasets with varying sample sizes | MSE | Helped |
| Provide additional data points as noise increases (ablation) | Do-PFN · Synthetic datasets with increasing noise | MSE | Helped |
| Increase Gaussian noise variance to two times the usual variance (ablation) | Do-PFN · Out-of-distribution synthetic data | MSE | Hurt |
| Use the pointwise average of training functions (ablation) | Do-PFN · Out-of-distribution synthetic data | MSE | Hurt |
| Use Laplace, T, Gumbel, or mixture noise instead of the training noise (ablation) | Do-PFN · Out-of-distribution synthetic data | MSE | No clear effect |
| Vary base-rate ATE magnitude (ablation) | Do-PFN · Synthetic CATE estimation | MSE | No clear effect |
Surprising findings
- Dont-PFN, a model trained on the same prior to predict observational outcomes, performs substantially worse than Do-PFN, indicating that the interventional pre-training objective matters.
- On the Common-Effect case study, standard regression models perform similarly to Do-PFN, since the treatment-covariate mismatch disappears when they are sampled independently.
Limitations
- Performance depends on how well the synthetic prior captures real-world causal structure; real-world validation is limited.
- Amortized approach lacks the statistical guarantees of traditional non-amortized estimators under known graphs.
- Only binary treatments and i.i.d. inputs are considered; counterfactuals and other intervention types are not covered.
Experimental setup
Datasets: synthetic SCM prior data RealCause Law School Admissions Amazon Sales
Benchmarks: Do-PFN synthetic case studies RealCause Law School Admissions Amazon Sales
Metrics: normalized MSE PICP critical difference (Nemenyi)
Baselines: TabPFN (v2) Random Forest Dont-PFN DoWhy (Int.) DoWhy (Cntf.) S-Learner T-Learner X-Learner DML DragonNet TARNet Causal Forest (DML)
Builds on: TabPFN prior-data fitted networks DoWhy EconML
Closest papers by meaning
- CausalPFN: Amortized Causal Effect Estimation via In-Context Learning (NeurIPS 2025 · similarity 0.89)
- MapPFN: Learning Causal Perturbation Maps in Context (NeurIPS 2026 · similarity 0.88)
- Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data (NeurIPS 2024 · similarity 0.87)
- CausalTab: Pretraining Across Causal Environments for Tabular Causal Discovery (NeurIPS 2026 · similarity 0.87)
- Learning Mixtures of Unknown Causal Interventions (NeurIPS 2024 · similarity 0.87)
- Towards Learning and Explaining Indirect Causal Effects in Neural Networks (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.