Research map / Learning theory and algorithms / Prediction and decision losses
Effect Size Estimation for Duration Recommendation in Online Experiments: Leveraging Hierarchical Models and Objective Utility Approaches
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes two data-driven methods to recommend the assumed effect size (AES) for online experiment duration: a three-layer heteroscedastic Gaussian mixture model fit by EM, and a utility-maximization approach balancing experiment cost and decision accuracy.
Problem
Choosing the assumed effect size for sample size and duration determination in large-scale online A/B experiments is usually done manually by experimenters, which does not scale to services running many experiments; naive pooled estimates underestimate AES because many experiments are flat or negati
Approach
A three-layer heteroscedastic GMM models latent positive/flat/negative experiment clusters with known per-experiment variances, estimating the mean of the positive component as AES; a second approach picks the AES that maximizes an expected utility combining experiment opportunity cost, in-experiment impact, and launch impact.
Key results
- In simulation with 200 experiments, the three-layer GMM achieved MSE 0.003 and MAE 0.137 versus 0.017/0.180 for two-layer GMM and 0.709/0.842 for pooled MLE.
- On 3,300 real Amazon experiments, utility-maximization obtained the highest average reward (3.41 x10^4) versus 3.08 for Pooled-MLE, 3.31 for two-layer GMM, and 3.28 for three-layer GMM.
- Pooled-MLE had the lowest empirical false positive and false negative rates in the real-data analysis but required the longest duration (4.0 weeks).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Three-layer GMM instead of Pooled-MLE to estimate AES | AES estimation · Simulated AES estimation | MSE: 0.709 -> 0.003 | Helped |
| Use Three-layer GMM instead of Two-layer GMM to estimate AES | AES estimation · Simulated AES estimation | MSE: 0.017 -> 0.003 | Helped |
| Use Utility-maximization instead of the highest-reward GMM baseline | AES recommendation · Meta-analysis results | Avg Reward (D): 2.01 -> \textbf{2.2} | Helped |
| Use Utility-maximization instead of the strongest baseline | AES recommendation · Amazon experiments | Avg Reward ($10^4$): 3.31 -> \textbf{3.41} | Helped |
| Use Utility-maximization instead of Pooled-MLE | AES recommendation · Amazon experiments | Empirical False Negative (%): 0.1\% -> 0.37\% | Hurt |
Surprising findings
- The pooled MLE, which is standard practice, underestimates AES severely (0.05% vs ground truth 2 in simulation), leading to overly long experiments.
- Utility-maximization yields the highest reward on real data even though the three-layer GMM is more accurate in simulation, showing accuracy and utility are not aligned.
Limitations
- Real dataset is confidential and cannot be shared, limiting reproducibility.
- Ground-truth effect size is unavailable in real data; the last-week observed effect is used as a heuristic proxy.
- Analysis restricted to a one-size-fits-all AES and a fixed 4-week maximum duration; personalized AES not explored.
Experimental setup
Datasets: 3,300 historical Amazon online experiments (confidential) simulated observed effect sizes from three-layer heteroscedastic GMM simulated experiment trajectories with beta-geometric weekly sample sizes
Benchmarks: simulated effect size estimation (MSE/MAE) Amazon real experiments meta-analysis (3,300 experiments) simulated trajectories
Metrics: MSE MAE empirical false positive rate empirical false negative rate average opportunity cost average launch impact average reward
Baselines: Pooled-MLE Two-layer GMM
Builds on: random-effect meta-analysis for AES two-layer Gaussian mixture model Welch's t-test power analysis
Closest papers by meaning
- On Online Experimentation without Device Identifiers (ICML 2024 · similarity 0.84)
- Efficient Adaptive Experimentation with Noncompliance (NeurIPS 2025 · similarity 0.84)
- Online Experimental Design With Estimation-Regret Trade-off Under Network Interference (NeurIPS 2025 · similarity 0.83)
- Benchmarking Estimators for Natural Experiments: A Novel Dataset and a Doubly Robust Algorithm (NeurIPS 2024 · similarity 0.83)
- Integrating Inference and Experimental Design for Contextual Behavioral Model Learning (AAAI 2025 · similarity 0.82)
- Optimistic Algorithms for Adaptive Estimation of the Average Treatment Effect (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.