Research map / Probabilistic and scientific modeling / Treatment effects
Do Contemporary Causal Inference Models Capture Real-World Heterogeneity? Findings from a Large-Scale Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks 16 CATE estimators on 43,200 observational-sampling variants of 12 real-world RCT datasets, using a new unbiased MSE-ranking statistic Q-hat, and finds most CATE estimates fail to beat trivial baselines.
Problem
Evaluating whether contemporary CATE estimation models capture real-world heterogeneity, given that ground-truth CATE is unobservable and existing evaluations rely on simulated outcomes or proxy losses with self-serving bias.
Approach
Applies observational sampling to real RCT data to create biased training subsets and held-out RCT evaluation sets, and introduces Q = MSE minus a constant with a sample statistic Q-hat that is unbiased for Q when propensity is known, so models can be ranked by MSE without counterfactual ground truth.
Key results
- 62% of fitted CATE estimators are degenerate (Q-hat >= 0, higher MSE than zero-effect predictor).
- Among datasets with a non-degenerate constant-effect baseline, only 20% of CATE estimators outperform the dml.lasso constant-effect model.
- Orthogonality-based learners (dml, r, dr, cforest) have a 71% average degenerate rate and win only 30% of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use contemporary CATE estimates instead of the zero-effect predictor | 16 CATE models · 43,200 observational-sampling datasets | MSE relative to zero-effect predictor: 62% | Hurt |
| Use a CATE estimator instead of the constant-effect estimator | CATE estimators · 25,440 datasets with non-degenerate constant-effect estimate | MSE relative to constant-effect estimator: 80% | Hurt |
| Use orthogonality-based learners rather than other learners | dml, r, dr, and cforest models · 43,200 observational-sampling datasets | win share: 30% | Hurt |
| Use s.xgb.cv as the CATE estimator | CATE models · 43,200 observational-sampling datasets | win share: 25.5% | Mixed |
| Explicitly construct X·T interaction inputs for the S-learner | S-learner with Ridge Regression · 43,200 observational-sampling datasets | win share: 12.9% -> 11.1% | Hurt |
| Explicitly construct X·T interaction inputs for the S-learner | S-learner with XGBoost · 43,200 observational-sampling datasets | win share: 25.5% -> 6.4% | Hurt |
Surprising findings
- Most CATE estimators fail to beat a trivial zero-effect predictor, and 80% fail to beat a constant-effect model despite being designed to capture heterogeneity.
- Orthogonality-based models, widely assumed to be strongest, underperform meta-learners such as S-learner with XGBoost.
Limitations
- Only 16 CATE models and 12 datasets were evaluated, limited by resources; deep learning and Gaussian process models were excluded.
- Findings may not generalize beyond the selected RCT datasets and sampling designs.
- Degenerate-rate findings may partly reflect implementation or propensity-model choices (e.g., clipping, random forest propensity), as the authors acknowledge.
Experimental setup
Datasets: criteo ferman hillstrom sandercock nataid natarms natcity natcrime natdrug nateduc natenvir natfare
Benchmarks: Criteo Ferman Hillstrom Sandercock GSS RCT datasets
Metrics: Q-hat degeneracy rate win share average rank MRR (simulation ranking agreement)
Baselines: s.xgb.cv s.ridge.cv dragon.nn dml.lasso cforest dr.xgb.cv r.xgb.cv t.xgb.cv
Builds on: observational sampling Double Machine Learning causal forests DR-learner DragonNet
Closest papers by meaning
- Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation (ICLR 2024 · similarity 0.90)
- A Non-parametric Direct Learning Approach to Heterogeneous Treatment Effect Estimation under Unmeasured Confounding (NeurIPS 2024 · similarity 0.89)
- Estimating Heterogeneous Treatment Effects by Combining Weak Instruments and Observational Data (NeurIPS 2024 · similarity 0.89)
- Unveiling the Potential of Robustness in Selecting Conditional Average Treatment Effect Estimators (NeurIPS 2024 · similarity 0.88)
- Conformal Inference of Individual Treatment Effects Using Conditional Density Estimates (AAAI 2025 · similarity 0.88)
- AutoCATE: End-to-End, Automated Treatment Effect Estimation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.