Research map / Probabilistic and scientific modeling / Molecules and proteins
Datasets for Online Controlled Experiments
C. H. Bryan Liu, Ângelo Cardoso, Paul Couturier, Emma J. McCoy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use mSPRT adaptive-stopping verdict instead of Welch t-test at experiment end | Welch t-test · ASOS Metric 2 | number of experiments with spurious significance: 18 not-significant t-test results flagged significant by mSPRT | Hurt |
| use mSPRT instead of Welch t-test | Welch t-test · ASOS Metric 1 | disagreement in significance verdict: 2 -> 7 | Mixed |
| use mSPRT instead of Welch t-test | Welch t-test · ASOS Metric 4 | disagreement in significance verdict: 4 -> 11 | Mixed |
| use mSPRT instead of Welch t-test | Welch t-test · ASOS Metric 3 | disagreement in significance verdict: 6 -> 9 | Mixed |
| compare mSPRT to Bayesian Bayes-factor test over time | mSPRT · five random ASOS experiments | evolution of evidence over time | Mixed |
| analyze p-value distribution over experiments with Welch t-test | ASOS dataset · ASOS Digital Experiments Dataset | fraction with p < 0.05 | No clear effect |
Closest papers by meaning
- Effect Size Estimation for Duration Recommendation in Online Experiments: Leveraging Hierarchical Models and Objective Utility Approaches (AAAI 2024 · similarity 0.86)
- On Online Experimentation without Device Identifiers (ICML 2024 · similarity 0.83)
- Just Ramp-Up: Debiasing Regression-based Estimator for A/B Tests under Network Interference (NeurIPS 2026 · similarity 0.83)
- Efficient Policy Evaluation Across Multiple Different Experimental Datasets (NeurIPS 2024 · similarity 0.83)
- Combining Experimental and Historical Data for Policy Evaluation (ICML 2024 · similarity 0.83)
- Efficient Adaptive Experimentation with Noncompliance (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
