Research map / Probabilistic and scientific modeling / Molecules and proteins
Efficient Randomized Experiments Using Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hybrid AIPW (H-AIPW), which optimally combines the standard AIPW estimator from experimental data with AIPW estimators that plug in predictions from multiple LLMs, to gain precision while keeping valid inference.
Problem
Randomized experiments are costly and imprecise, while in silico experiments using foundation model predictions can yield invalid inference if the predictions are biased. The goal is to use foundation model predictions to improve ATE precision without losing validity or risking higher variance than
Approach
Treat each foundation-model outcome regression as an AIPW estimator and combine them, together with the standard experimental-data AIPW, via variance-minimizing convex weights estimated from the empirical covariance of influence functions. The combined estimator is asymptotically normal and has variance no larger than any component, even when the models are arbitrarily biased.
Key results
- H-AIPW achieved the lowest variance on most of the eight experiments, with reductions of roughly 5-11% versus standard AIPW at n=100 and n=200 (e.g., 10.39 vs 11.72 on Melin et al. at n=100).
- Empirical coverage of H-AIPW stayed near the nominal 95% level (e.g., 94.4% to 98.4% across studies and sample sizes).
- Reported sample-size savings of up to 20% to match standard AIPW precision; with 7 models at n=50, variance gain was about 16%, but adding models caused 2-4% undercoverage.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate multiple foundation-model predictions with standard AIPW | H-AIPW · Eight randomized experiments, n=100 and n=200 | variance: 11.72 -> 10.39 | Helped |
| Replace one foundation model with an ensemble of 5 to 7 models (ablation) | H-AIPW · Study by [cite], 50 samples | variance: approximately 6% to nearly 12%; around 16% | Helped |
| Increase the ensemble from 1 to 7 foundation models (ablation) | H-AIPW · Study by [cite], 50 samples | coverage: 2%–4% | Hurt |
| Average predictions over more inference-time prompts (ablation) | H-AIPW · Three randomized studies using GPT-4o | variance | Helped |
| Use larger language models rather than smaller models (ablation) | H-AIPW · LLM prediction evaluation across randomized experiments | MSE | Helped |
Surprising findings
- Adding more foundation models increases undercoverage at small sample sizes (n=50) due to covariance estimation error, requiring cross-fitting.
- PPCT was sometimes less precise than standard AIPW, and boosting-based AIPW had very high variance at small sample sizes.
Limitations
- Performance depends on the foundation models being accurate and aligned with the experimental domain.
- Finite-sample covariance estimation becomes unstable with many models relative to sample size, potentially causing undercoverage.
- Experiments are limited to social science survey experiments; image treatment evaluated with only GPT-4o.
Experimental setup
Datasets: TESS survey experiments (eight studies from Melin et al. 2022, Silverman et al. 2022, Kennedy et al. 2020, Fahey et al.
Benchmarks: TESS survey experiments (eight studies, subsampled at n=100 and n=200)
Metrics: variance (scaled, averaged over subsampling repetitions) coverage probability at 95% nominal level mean squared error of LLM predictions
Baselines: difference in means standard AIPW (linear) AIPW (boosting) PPCT PROCOVA
Builds on: augmented inverse probability weighting prediction-powered inference PPI++ PPCT PROCOVA
Closest papers by meaning
- Estimating Distributional Treatment Effects in Randomized Experiments: Machine Learning for Variance Reduction (ICML 2024 · similarity 0.86)
- Benchmarking Estimators for Natural Experiments: A Novel Dataset and a Doubly Robust Algorithm (NeurIPS 2024 · similarity 0.86)
- Efficient Adaptive Experimentation with Noncompliance (NeurIPS 2025 · similarity 0.86)
- On Efficient Estimation of Distributional Treatment Effects under Covariate-Adaptive Randomization (ICML 2025 · similarity 0.85)
- Prediction-Powered Causal Inferences (NeurIPS 2025 · similarity 0.85)
- Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choice (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.