Research map / Learning theory and algorithms / Prediction and decision losses
Benchmarking Estimators for Natural Experiments: A Novel Dataset and a Doubly Robust Algorithm
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the RORCO natural-experiment dataset (early childhood literacy), a benchmark of 20+ treatment effect estimators on semi-synthetic outcomes, a variance analysis of split-trained doubly robust estimators, and the Double-Double estimator.
Problem
Estimating average treatment effects from natural experiments where treatment assignment is non-random and correlated with outcomes and treatment effects; estimators give inconsistent results and there is no agreed best method or realistic benchmark.
Approach
Benchmark over 20 estimators on a semi-synthetic dataset whose outcomes are designed with domain experts, and derive the exact variance of split-trained doubly robust estimators, motivating Double-Double, which uses inverse-propensity-based sample weights when fitting outcome regressions.
Key results
- On the semi-synthetic RORCO benchmark, the Doubly Robust estimator achieves mean squared error 9.98e-07 versus 1.07e-05 for Double-Double and 1.92e-04 for TARNet (best CATENet).
- Across 20+ estimators on RORCO Real, estimates of the normalized treatment effect range from -0.607 to 0.459, showing inconsistent results on real data.
- Exact variance expression for split-trained doubly robust estimators is derived, and it motivates Double-Double with weights (1-p)/p^2 and p/(1-p)^2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add weighting to the doubly robust estimator (ablation) | Doubly Robust on RORCO · RORCO | squared error (mean): 9.98e-07 -> 4.02e-06 | Hurt |
| Add double weighting to the doubly robust estimator (ablation) | Doubly Robust on RORCO · RORCO | squared error (mean): 9.98e-07 -> 3.80e-06 | Hurt |
| Add split training to the doubly robust estimator (ablation) | Doubly Robust on RORCO · RORCO | squared error (mean): 9.98e-07 -> 9.82e-05 | Hurt |
| Add split training and weighting to the doubly robust estimator (ablation) | Doubly Robust on RORCO · RORCO | squared error (mean): 9.98e-07 -> 1.12e-04 | Hurt |
| Use double weighting rather than unweighted loss in the split-trained doubly robust estimator (ablation) | Doubly Robust on RORCO · RORCO | squared error (mean): 9.82e-05 -> 1.07e-05 | Helped |
| Use the standard doubly robust estimator rather than Double-Double | Treatment effect estimation on RORCO · RORCO | squared error (mean): 1.07e-05 -> 9.98e-07 | Helped |
Surprising findings
- Doubly robust estimators outperform complicated neural CATE estimators by orders of magnitude on the benchmark.
- Estimators applied to the real RORCO outcomes return substantially different treatment effect estimates, ranging from -0.607 to 0.459 in normalized units.
Limitations
- Observational RORCO Real relies on strong clinic-school proximity assumptions and class-level observations.
- Theoretical variance analysis assumes known propensity scores and applies only to split-trained doubly robust estimators.
- Synthetic outcomes reflect expert assumptions and may not match real-world behavior.
Experimental setup
Datasets: RORCO RORCO Real JOBS TWINS IHDP NEWS ACIC 2016 ACIC 2017
Benchmarks: RORCO RORCO Real ACIC 2016 ACIC 2017 IHDP JOBS NEWS TWINS
Metrics: squared error mean squared error variance runtime
Baselines: Regression Discontinuity Propensity Stratification Horvitz-Thompson TMLE Off-policy TARNet DragonNet FlexTENet TNet OffsetNet
Builds on: Horvitz-Thompson estimator Off-policy estimator (Mou et al.) CATENets TMLE
Closest papers by meaning
- Doubly robust identification of treatment effects from multiple environments (ICLR 2025 · similarity 0.89)
- Efficient Randomized Experiments Using Foundation Models (NeurIPS 2025 · similarity 0.86)
- End-To-End Causal Effect Estimation from Unstructured Natural Language Data (NeurIPS 2024 · similarity 0.86)
- A Non-parametric Direct Learning Approach to Heterogeneous Treatment Effect Estimation under Unmeasured Confounding (NeurIPS 2024 · similarity 0.86)
- Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choice (ICML 2024 · similarity 0.86)
- Unveiling the Potential of Robustness in Selecting Conditional Average Treatment Effect Estimators (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.