Research map / Probabilistic and scientific modeling / Treatment effects
Adjustment for Confounding using Pre-Trained Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies when latent features from pre-trained neural networks can serve as valid confounding adjustment for ATE estimation via DML, showing neural networks achieve fast rates by adapting to intrinsic dimension and hierarchical sparsity.
Problem
Estimating average treatment effects under confounding from non-tabular data (images, text) using latent features from pre-trained networks, where the representations are non-identifiable up to invertible linear transformations and high-dimensional.
Approach
Shows that smoothness and low intrinsic dimension are invariant under invertible linear transformations of representations, whereas additivity and sparsity are not; combines manifold structure with a hierarchical composition model so neural-network nuisance estimators attain fast rates enabling root-n valid DML inference.
Key results
- On IMDb label confounding, DML with linear nuisance estimators yields unbiased ATE estimates with good CI coverage, while lasso and random-forest DML and S-Learner show strong bias.
- With complex confounding on X-ray data, neural-network DML shows much less bias and high coverage, the only estimator enabling valid inference (S-Learner CIs too optimistic).
- Estimated intrinsic dimension of DenseNet-121 X-ray representations is far below ambient dimension 1024 (about 12 for MLE-type estimates).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Leave confounding unadjusted | Naive ATE estimator · IMDb label confounding | ATE bias | Hurt |
| Use S-Learner with pre-trained representations | S-Learner · IMDb label confounding | ATE bias | Hurt |
| Use L1-penalized linear nuisance estimators | DML with pre-trained representations · IMDb label confounding | ATE bias | Hurt |
| Use random-forest nuisance estimators | DML with pre-trained representations · IMDb label confounding | ATE bias | Hurt |
| Use unpenalized linear nuisance estimators | DML with BERT representations · IMDb label confounding | ATE bias | Helped |
| Use unpenalized linear nuisance estimators | DML with BERT representations · IMDb label confounding | CI coverage | Helped |
| Use unpenalized linear nuisance estimators | DML with Densenet-121 representations · X-ray label confounding | ATE bias | Helped |
| Use neural-network nuisance estimators | DML · Complex confounding | ATE bias | Helped |
| Use DML rather than S-Learner with neural-network nuisance estimation | Neural-network nuisance estimators · Complex confounding | CI coverage | Helped |
| Use S-Learner | Complex confounding estimators · Complex confounding | CI calibration | Hurt |
| Use pre-trained feature extractors instead of CNNs trained from scratch | DML · X-ray label confounding | ATE bias | Helped |
| Use pre-trained feature extractors instead of CNNs trained from scratch | DML · X-ray label confounding | CI calibration | Helped |
Surprising findings
- Lasso and random-forest nuisance estimators in DML fail despite being standard, because representations are only identifiable up to rotations, which destroys sparsity.
- The S-Learner with neural networks yields overly optimistic confidence intervals even while DML remains calibrated.
Limitations
- Focuses on a single confounding modality; multi-modal confounding is left to future work.
- Studies only the ATE; ATT and CATE are not analyzed.
- Experiments use subsampled data (1,000 and 4,000 IMDb representations; 3,769 X-rays) and simulated treatment and outcomes.
Experimental setup
Datasets: IMDb Movie Reviews Guangzhou pediatric chest X-ray dataset
Metrics: bias coverage of 95% confidence intervals point estimates of ATE
Baselines: Naive estimator Oracle estimator S-Learner DML with lasso DML with random forest DML with linear nuisance
Builds on: double machine learning hierarchical composition model manifold hypothesis
Closest papers by meaning
- Bounds on Representation-Induced Confounding Bias for Treatment Effect Estimation (ICLR 2024 · similarity 0.88)
- Learning Representations of Instruments for Partial Identification of Treatment Effects (ICML 2025 · similarity 0.86)
- Do-PFN: In-Context Learning for Causal Effect Estimation (NeurIPS 2025 · similarity 0.86)
- Identifying Representations for Intervention Extrapolation (ICLR 2024 · similarity 0.85)
- A Neural Framework for Generalized Causal Sensitivity Analysis (ICLR 2024 · similarity 0.85)
- ODE Discovery for Longitudinal Heterogeneous Treatment Effects Inference (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.