Research map / Probabilistic and scientific modeling / Treatment effects
Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a weighted two-armed bandit (WTAB) test statistic with doubly robust counterfactual estimation, ML nuisance models, and Cauchy-combined permutations (PWTAB) to boost power for detecting minor average treatment effects in A/B tests.
Problem
Detecting minor average treatment effects in large-scale A/B tests, where normal-based or expanded test statistics lack sensitivity to small effects.
Approach
Breaks data exchangeability via a maximum-tail-probability two-armed bandit policy over a weighted mean-volatility statistic, with DR counterfactual estimates from LightGBM/XGBoost stacking under double machine learning, and aggregates p-values over permutations via Cauchy combination.
Key results
- Under the null, PWTAB, WTAB, z-DML, CUPED and DIM all keep type I error near 0.05 across simulation settings (e.g., 0.046 to 0.060).
- On semi-synthetic data, PWTAB (stacking) reaches power 0.764 versus 0.740 for CUPED and 0.498 for DIM with type I error 0.052.
- On real dataset B, PWTAB with stacking gives p=0.032 where CUPED (0.053) and z-DML (0.053) fail to reject at 0.05.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Aggregate permuted WTAB p-values with Cauchy combination rather than using WTAB without permutation aggregation (ablation) | WTAB · Real-world-derived synthetic data, LightGBM | power: 0.738 -> 0.758 | Helped |
| Aggregate permuted WTAB p-values with Cauchy combination rather than using WTAB without permutation aggregation (ablation) | WTAB · Real-world-derived synthetic data, Stacking | power: 0.732 -> 0.764 | Helped |
| Use PWTAB instead of z-DML | A/B testing with LightGBM counterfactual estimation · Real-world-derived synthetic data, LightGBM | power: 0.744 -> 0.758 | Helped |
| Use PWTAB instead of CUPED | A/B testing with LightGBM counterfactual estimation · Real-world-derived synthetic data, LightGBM | power: 0.740 -> 0.758 | Helped |
| Use PWTAB instead of DIM | A/B testing with LightGBM counterfactual estimation · Real-world-derived synthetic data, LightGBM | power: 0.498 -> 0.758 | Helped |
| Use PWTAB rather than WTAB (ablation) | A/B testing on dataset B · Dataset B | p-value: 0.043 -> 0.044 | No clear effect |
| Use PWTAB rather than CUPED | A/B testing on dataset B · Dataset B | p-value: 0.053 -> 0.044 | Helped |
| Use PWTAB rather than DIM | A/B testing on dataset B · Dataset B | p-value: 0.228 -> 0.044 | Helped |
| Use PWTAB instead of WTAB (ablation) | A/B testing with LightGBM counterfactual estimation · Real-world-derived synthetic data, LightGBM | Type I error: 0.052 -> 0.052 | No clear effect |
| Use stacking instead of LightGBM for nuisance-function estimation | PWTAB · Real-world-derived synthetic data | power: 0.758 -> 0.764 | Helped |
| Increase the number of permutations beyond 25 (ablation) | PWTAB · A/B testing | power | No clear effect |
Surprising findings
- WTAB and the z-test show comparable power in most settings, which the authors attribute to the one-sided versus two-tailed rejection design.
Limitations
- Relies on the correctness of at least one nuisance model (double robustness) and ML convergence rates; the authors note tuning of lambda is needed to control type I error.
- Real-data results come from a proprietary ride-sharing company, so they are not reproducible; the paper also notes ML estimates are less refined than explicit models.
Experimental setup
Datasets: synthetic simulation data (n=20000) three real-world ride-sharing RCT datasets (A, B, C) semi-synthetic dataset from real-world data
Benchmarks: synthetic A/B simulations (32 settings) real ride-sharing datasets B and C semi-synthetic dataset
Metrics: type I error rate statistical power p-value variance reduction
Baselines: WTAB z-DML CUPED DIM
Builds on: strategic central limit theorem Double/Debiased Machine Learning Cauchy combination test CUPED
Closest papers by meaning
- Asymptotically Optimal and Computationally Efficient Average Treatment Effect Estimation in A/B testing (ICML 2024 · similarity 0.86)
- Just Ramp-Up: Debiasing Regression-based Estimator for A/B Tests under Network Interference (NeurIPS 2026 · similarity 0.85)
- Simulation-Based Inference for Adaptive Experiments (NeurIPS 2025 · similarity 0.85)
- On Universally Optimal Algorithms for A/B Testing (ICML 2024 · similarity 0.85)
- Peeking with PEAK: Sequential, Nonparametric Composite Hypothesis Tests for Means of Multiple Data Streams (ICML 2024 · similarity 0.84)
- Optimistic Algorithms for Adaptive Estimation of the Average Treatment Effect (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.