Research map / Probabilistic and scientific modeling / Treatment effects
Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks 34 surrogate metrics for CATE model selection across 78 datasets and 415 estimators, using AutoML-tuned nuisance models, a two-level selection strategy, and causal ensembling.
Problem
Model selection for conditional average treatment effect (CATE) estimators without access to counterfactual outcomes; prior surrogate metric evaluations were limited in datasets, baselines, and nuisance-model tuning.
Approach
Fair, large-scale comparison of surrogate metrics with nuisance models tuned by AutoML (FLAML), plus new metrics (DR/TMLE switching and blending, TMLE score, calibration and Qini variants), a two-level selection strategy that first selects within each meta-learner class, and softmax-weighted causal ensembling; realistic benchmarks generated with RealCause.
Key results
- DR-based and TMLE T-score variants are dominating across all datasets under the single-level strategy (e.g., DR T Score 0.56 on ACIC 2016 vs S Score 0.95).
- T Score (plug-in) with AutoML-tuned nuisances is rarely dominated and is competitive with DR variants.
- The two-level selection strategy outperforms single-level in about 28.7% of cases and is never dominated by it; causal ensembling is statistically better in about 5.8% of experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DR/TMLE-based surrogate metrics instead of other surrogate metrics | CATE model selection · All evaluated datasets | Normalized PEHE | Helped |
| Use the T-Learner rather than the S-Learner to construct the score (ablation) | CATE surrogate metrics · ACIC 2016, LaLonde CPS, LaLonde PSID, and TWINS | Normalized PEHE | Helped |
| Add adaptive propensity switching to DR T Score (ablation) | DR T Score · All evaluated datasets | Normalized PEHE | No clear effect |
| Add continuous adaptive blending to DR T Score (ablation) | DR T Score · All evaluated datasets | Normalized PEHE | No clear effect |
| Use two-level selection instead of single-level selection (ablation) | CATE model selection · All datasets and metrics | Normalized PEHE: 28.7% | Helped |
| Ensemble selected CATE estimators rather than selecting one estimator (ablation) | Two-level CATE model selection · All datasets and metrics | Normalized PEHE: 5.8% | Helped |
| Use a T-Learner-based plug-in metric instead of an S-Learner-based plug-in metric (ablation) | Plug-in surrogate metrics · ACIC 2016, LaLonde CPS, LaLonde PSID, and TWINS | Normalized PEHE | Helped |
Surprising findings
- Plug-in T/X Score with AutoML-tuned nuisance models is rarely dominated, contrary to prior benchmarks that found plug-in metrics sub-optimal.
- Qini DR score ranks among the worst under single-level selection on TWINS but becomes dominating for most datasets under the two-level strategy.
Limitations
- Realistic benchmarks rely on only three datasets generated with RealCause, so generalization is uncertain.
- Two-level strategy's choice of meta-learner metric is heuristic and not guaranteed optimal.
- Improvements from propensity-clipping variants over DR are not statistically significant.
Experimental setup
Datasets: ACIC 2016 LaLonde CPS LaLonde PSID Twins
Benchmarks: ACIC 2016 LaLonde CPS LaLonde PSID TWINS
Metrics: normalized PEHE PEHE of selected estimators
Baselines: S Score T Score X Score DR Score R Score Influence Score Value Score Value DR Score Match Score IPW Score
Builds on: R-Learner DR-Learner FLAML RealCause Super Learner
Closest papers by meaning
- Do Contemporary Causal Inference Models Capture Real-World Heterogeneity? Findings from a Large-Scale Benchmark (ICLR 2025 · similarity 0.90)
- Unveiling the Potential of Robustness in Selecting Conditional Average Treatment Effect Estimators (NeurIPS 2024 · similarity 0.90)
- AutoCATE: End-to-End, Automated Treatment Effect Estimation (ICML 2025 · similarity 0.88)
- Local Learning for Covariate Selection in Nonparametric Causal Effect Estimation with Latent Variables (NeurIPS 2025 · similarity 0.87)
- Continuous Treatment Effects with Surrogate Outcomes (ICML 2024 · similarity 0.87)
- A Non-parametric Direct Learning Approach to Heterogeneous Treatment Effect Estimation under Unmeasured Confounding (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.