Research map / Probabilistic and scientific modeling / Treatment effects
Smoke and Mirrors in Causal Downstream Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that common ML design choices (annotation sampling, frozen encoder choice, discretizing predictions, accuracy-based model selection) can bias treatment effect estimates from high-dimensional RCT data, and releases the ISTAnt ant-grooming benchmark and a CausalMNIST synthetic benchmark.
Problem
Estimating average treatment effects from high-dimensional observations (e.g., video frames of an RCT) with deep learning-annotated outcomes; standard ML practices may yield biased causal estimates even when predictive accuracy is high.
Approach
Formalizes a treatment effect bias (TEB) for ML-predicted outcomes, proves that accuracy bounds TEB and that discretizing predictions biases ATE, and empirically shows that TEB on a validation set (not accuracy) is the better model-selection signal, using frozen pretrained ViT encoders with MLP heads on the new ISTAnt dataset.
Key results
- Across 6,480 fine-tuned heads, discretizing predictions significantly worsens ATE estimation (t = -10.42, p = 1.07e-25).
- Above balanced accuracy 0.9, TERB does not correlate with accuracy; models with balanced accuracy above 0.95 still show TERB up to ±50%.
- Selecting models by validation TEB rather than accuracy underestimates the effect by 11% (few-shot, random) and 18% (few-shot, experiment) rather than the accuracy-selected model's bias.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use per-video random annotation instead of experiment- or position-based annotation | ISTAnt fine-tuned models · ISTAnt few-shot | absolute treatment effect bias | Helped |
| Use per-video random annotation instead of experiment-based annotation | ISTAnt fine-tuned models · ISTAnt many-shot | treatment effect bias | Mixed |
| Discretize model predictions at 0.5 | ISTAnt fine-tuned models · ISTAnt | absolute treatment effect bias: t statistic=-10.42, p-value=$1.07\cdot 10^{-25}$ | Hurt |
| Select the best model using validation treatment effect bias rather than accuracy | ISTAnt fine-tuned models · ISTAnt few-shot | ATE estimation error: 11% | Helped |
| Select the best model using validation treatment effect bias rather than accuracy | ISTAnt fine-tuned models · ISTAnt experiment sampling | ATE estimation error: 18% | Helped |
| Use classification accuracy as a proxy for treatment effect bias | ISTAnt fine-tuned models · ISTAnt | treatment effect bias | No clear effect |
| Use MAE instead of the other evaluated encoders | ISTAnt fine-tuned models · ISTAnt | downstream predictive performance | Hurt |
| Use random annotation instead of biased annotation | CausalMNIST ConvNet · CausalMNIST few-shot | treatment effect bias: 1.607, 0.111; 3.911, 1.7 $\cdot 10^{-4}$ | Helped |
| Discretize model predictions at 0.5 | CausalMNIST ConvNet · CausalMNIST | absolute treatment effect bias | Hurt |
Surprising findings
- Higher classification accuracy does not imply less treatment effect bias; accuracy is not a proxy for causal estimation quality.
- Discretizing sigmoid outputs into hard labels significantly biases the ATE, even when the underlying classifier is consistent.
Limitations
- Backbones were kept frozen; no debiasing training was attempted.
- Unlabelled data was not used for semi-supervised training.
- Real-world ATE ground truth relies on expert annotations and the RCT's design, so the true causal effect is not known.
Experimental setup
Datasets: ISTAnt CausalMNIST MNIST
Benchmarks: ISTAnt CausalMNIST
Metrics: treatment effect relative bias (TERB) treatment effect bias (TEB) accuracy balanced accuracy binary cross-entropy Spearman rank correlation Fréchet distance
Baselines: ViT-B ViT-L CLIP-ViT-B CLIP-ViT-L MAE DINOv2
Builds on: Average treatment effect framework MNIST Adam
Closest papers by meaning
- The third pillar of causal analysis? A measurement perspective on causal representations (NeurIPS 2025 · similarity 0.87)
- Bayesian Active Learning for Bivariate Causal Discovery (ICML 2025 · similarity 0.87)
- Estimating Interventional Distributions with Uncertain Causal Graphs through Meta-Learning (NeurIPS 2025 · similarity 0.87)
- Higher-Order Causal Message Passing for Experimentation with Complex Interference (NeurIPS 2024 · similarity 0.87)
- Prediction-Powered Causal Inferences (NeurIPS 2025 · similarity 0.87)
- Towards Causal Foundation Model: on Duality between Optimal Balancing and Attention (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.