Research map / Learning theory and algorithms / Prediction and decision losses
Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-stage RCT design that screens arms in a first uniform stage, then certifies a high-probability lower bound on the best retained arm's effect in a second uniform stage, with sample-splitting and Bayesian prior variants.
Problem
Single-stage RCTs waste samples on suboptimal arms, reducing the power of high-probability lower-bound certificates on treatment effects, while fully adaptive trials are logistically hard to deploy.
Approach
Restrict adaptivity to two uniform-allocation stages: screen arms in stage one (top-k policy chosen via sample splitting, or greedy submodular selection under a prior), then compute a Hoeffding-based lower bound on the retained arm using stage-two data.
Key results
- Sample split two-stage design improves over single-stage by about 7% when first stage is 30% of budget (synthetic).
- Sample split is within 0.5% of the omniscient top-k policy at T=40000 and within 3% of UCB at best first-stage size.
- Prior-based two-stage policy beats UCB by 23% on the semi-synthetic gerontology setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use two-stage sample splitting instead of a single-stage RCT | RCT design · Synthetic RCT, s1=30% of budget | normalized certificate: 7% | Helped |
| Use two-stage sample splitting instead of a single-stage RCT | RCT design · Synthetic RCT, s1=70% of budget | normalized certificate: 1% | Helped |
| Increase the total budget to T=40000 (ablation) | Sample Split · Synthetic RCT, T=40000 | normalized certificate: within 0.5% | No clear effect |
| Use Best Arm (k=1) instead of Sample Split | Two-stage RCT · Synthetic RCT, T=40000 | normalized certificate: 3% | Helped |
| Use Best Arm (k=1) instead of Sample Split | Two-stage RCT · Synthetic RCT, T=1000 | normalized certificate: 13% | Hurt |
| Use two-stage sample splitting instead of two-stage Thompson Sampling | RCT design · Synthetic RCT | normalized certificate: within 1.5% | No clear effect |
| Use two-stage sample splitting instead of UCB | RCT design · Synthetic RCT, s1=30% of budget | normalized certificate: within 3% | Hurt |
| Add an informative prior with beta=4 | Prior-based two-stage policy · Synthetic RCT, beta=4 | normalized certificate: 18% | Helped |
| Use prior-based methods instead of UCB | RCT design · Gerontology semi-synthetic RCT | normalized certificate: 23% | Helped |
| Use sample splitting instead of UCB among prior-free methods | RCT design · Gerontology semi-synthetic RCT | normalized certificate: within 2% | No clear effect |
| Increase prior misspecification through directional Gaussian noise (ablation) | Prior-based policy · Synthetic RCT with prior misspecification | normalized certificate | Hurt |
| Use more than two stages while using only the last stage for the certificate (ablation) | Multi-stage RCT · Synthetic RCT | certificate | Hurt |
Surprising findings
- Best-arm (k=1) two-stage policy beats sample splitting by 3% at large budget T=40000, indicating the adaptive selection of k is not always better.
- Prior-based policies are not robust to prior misspecification and degrade as misspecification grows.
Limitations
- Theoretical optimality of top-k policies requires first-order stochastic dominance assumption.
- Prior-based methods are sensitive to misspecification of the prior.
- Certificate analysis uses Hoeffding-type bounds and the approximation guarantees are only sketched in places.
Experimental setup
Datasets: synthetic Bernoulli arms with uniform or beta-distributed means gerontology meta-analysis effect sizes (75 values, semi-synthetic)
Benchmarks: synthetic RCT simulations (n=10, delta=0.1) semi-synthetic gerontology effect sizes
Metrics: normalized certificate (ratio of lower bound to max mean) certificate spread
Baselines: single-stage uniform RCT two-stage random k two-stage best arm two-stage successive elimination two-stage Thompson sampling successive elimination UCB
Builds on: Hoeffding's inequality successive elimination Thompson sampling UCB
Closest papers by meaning
- Adaptive Data-Borrowing for Improving Treatment Effect Estimation using External Controls (NeurIPS 2025 · similarity 0.87)
- Using Surrogates in Covariate-adjusted Response-adaptive Randomization Experiments with Delayed Outcomes (NeurIPS 2024 · similarity 0.86)
- Enhancing Statistical Validity and Power in Hybrid Controlled Trials: A Randomization Inference Approach with Conformal Selective Borrowing (ICML 2025 · similarity 0.85)
- Safe Exploration in Dose Finding Clinical Trials with Heterogeneous Participants (ICML 2024 · similarity 0.85)
- Generalizing Causal Effects from Randomized Controlled Trials to Target Populations across Diverse Environments (ICML 2025 · similarity 0.83)
- Stronger Neyman Regret Guarantees for Adaptive Experimental Design (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.