Research map / Learning theory and algorithms / Prediction and decision losses
YEAST: Yet Another Sequential Test
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes YEAST, a continuous-monitoring sequential test for count and real-valued A/B metrics that uses a constant boundary set from a robust variance estimate at a prespecified event horizon, with asymptotic Type-I error control and evaluations on simulations and semi-synthetic online retail data.
Problem
Sequential A/B testing of machine learning models needs continuous monitoring with false-detection control, but existing sequential tests either limit the number of interim analyses or lose statistical power, and repeated-user dependence complicates variance estimation.
Approach
Track the cumulative control-minus-treatment metric difference over events and compare it with a constant boundary z_{1-alpha/2} * sqrt(N * V_N), where V_N is a cluster-robust variance estimate at the prespecified horizon N; the asymptotic guarantee comes from a maximal inequality plus a terminal CLT and future sign-balance conditions, allowing some repeated-user dependence.
Key results
- On the semi-synthetic retail study with robust variance, YEAST's empirical false detection rate was 0.044, within one point of the 5% nominal level, while the non-robust variance version reached 0.168.
- In the full-period retail power study at a 20% order-value decrease, YEAST detected 0.676 of replications, close to the non-sequential z-test (0.685) and above LanDeMetsOBF (0.585) and mSPRT/GAVI variants.
- In synthetic normal simulations at effect size 0.1, YEAST achieved power 0.439 versus 0.216-0.237 for mSPRT and GAVI variants at a controlled false detection rate of 0.047.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use clustered robust rather than non-robust variance estimation (ablation) | YEAST on Online Retail · Online Retail, null | false detection rate: 0.168 ± 0.004 -> 0.044 ± 0.002 | Helped |
| use YEAST rather than LanDeMetsOBF | Online Retail sequential testing · Online Retail, 5%, 10%, and 20% decrease | power: 0.075 ± 0.003 -> 0.114 ± 0.004; 0.181 ± 0.004 -> 0.252 ± 0.005; 0.585 ± 0.005 -> 0.676 ± 0.005 | Helped |
| overestimate NV_N by 10% (ablation) | YEAST · Normal simulation, effects 0.1 and 0.2 | power: 0.439 ± 0.005 -> 0.398 ± 0.005; 0.918 ± 0.003 -> 0.901 ± 0.003 | Hurt |
| overestimate NV_N by 20% (ablation) | YEAST · Normal simulation, effects 0.1 and 0.2 | power: 0.439 ± 0.005 -> 0.362 ± 0.005; 0.918 ± 0.003 -> 0.884 ± 0.003 | Hurt |
| underestimate NV_N by 20% (ablation) | YEAST · Normal simulation, null | false detection rate: 0.047 ± 0.002 -> 0.074 ± 0.003 | Hurt |
| use YEAST instead of eligible martingale-based methods | Sequential tests · Normal simulation, effects 0.1 and 0.2 | power: 0.439 ± 0.005 vs 0.235 ± 0.004; 0.918 ± 0.003 vs 0.756 ± 0.005 | Helped |
| use YEAST rather than the fixed-horizon Classical z-test | Online Retail · Online Retail, 5%, 10%, 20%, and 50% decrease | power: 0.114 ± 0.004 vs 0.114 ± 0.004; 0.252 ± 0.005 vs 0.255 ± 0.005; 0.676 ± 0.005 vs 0.685 ± 0.005; 1.000 ± 0.000 vs 1.000 ± | No clear effect |
| use YEAST rather than GSTphi3 | Discrete sequential monitoring · Effect 0.2 standard deviations, 14 checks | power: 0.93 -> 0.90 | Hurt |
Surprising findings
- Using the non-robust variance estimate (ignoring repeated-user dependence) inflated the false detection rate of YEAST to 0.168 in the retail study.
- SeqC2ST-QDA had a null false detection rate of 0.242, far above nominal level.
Limitations
- Strong null (no effect on event-generation process) causes sensitivity to treatments that change variance or offsetting multi-effect changes with neutral mean.
- Requires a prespecified event horizon N and a variance estimate from pre-experiment data.
- Assumptions 2-3 are not verified for the retail data; results do not establish uniform superiority over other sequential methods.
Experimental setup
Datasets: Online Retail dataset (UK online store, 21,269 orders used)
Benchmarks: semi-synthetic Online Retail A/B replications synthetic normal/non-normal A/B simulations
Metrics: false detection rate power sample savings
Baselines: mSPRT GAVI LanDeMetsOBF SeqC2ST-QDA Bonferroni GST
Builds on: mSPRT group sequential testing with alpha spending O'Brien-Fleming boundary GAVI
Closest papers by meaning
- Sequential Kernel Goodness-of-fit Testing (ICML 2024 · similarity 0.86)
- Strategic A/B testing via Maximum Probability-driven Two-armed Bandit (ICML 2025 · similarity 0.83)
- Peeking with PEAK: Sequential, Nonparametric Composite Hypothesis Tests for Means of Multiple Data Streams (ICML 2024 · similarity 0.83)
- WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales (ICML 2025 · similarity 0.82)
- Learning to Discover at Test Time (ICML 2026 · similarity 0.82)
- Prediction-Powered Active Testing (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.