Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Model Selection for Production System via Automated Online Experiments
Zhenwen Dai, Praveen Chandar, Ghazal Fazelnia, Benjamin Carterette, Mounia Lalmas
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AOE instead of BO on recorded accumulative metrics | AOE · Classification (letter) | metric gap: 0.059 -> 0.0029 | Improved |
| Use AOE instead of IS-EI | AOE · Classification (letter) | metric gap: 0.042 -> 0.0029 | Improved |
| Use AOE instead of DR-EI (strongest baseline) | AOE · Classification (letter) | metric gap: 0.013 -> 0.0029 | Improved |
| Use AOE instead of DR-g | AOE · Classification (letter) | RMSE: 0.054 -> 0.011 | Improved |
| Use AOE instead of BO for estimating accumulative metrics | AOE · Classification (letter) | RMSE: 1.31 -> 0.011 | Improved |
| Use doubly robust instead of importance sampling OPE estimator | IS-g / IS-EI baselines · Classification (letter) | metric gap: 0.042 -> 0.013 | Improved |
| Add exploration (EI) on top of IS greedy selection | IS-g · Classification (letter) | metric gap: 0.024 -> 0.042 | Hurt |
| Add exploration (EI) on top of DR greedy selection | DR-g · Classification (letter) | metric gap: 0.020 -> 0.013 | Improved |
| Swap acquisition function among EI, PI, UCB (ablation) | AOE · Classification (letter) | metric gap and RMSE | No clear effect |
| Use AOE instead of DR-EI | AOE · Recommender (MovieLens 100k) | metric gap: 0.0063 -> 0. | Improved |
| Use AOE instead of BO | AOE · Recommender (MovieLens 100k) | RMSE: 0.715 -> 0.016 | Improved |
| Use AOE instead of DR-EI | AOE · Recommender (MovieLens 100k) | RMSE: 0.085 -> 0.016 | Improved |
Closest papers by meaning
- Offline Model-Based Optimization by Learning to Rank (ICLR 2025 · similarity 0.82)
- Quilt: Robust Data Segment Selection against Concept Drifts (AAAI 2024 · similarity 0.82)
- Offline Model-Based Optimization via Policy-Guided Gradient Search (AAAI 2024 · similarity 0.82)
- Model Assessment and Selection under Temporal Distribution Shift (ICML 2024 · similarity 0.82)
- Diversified Batch Selection for Training Acceleration (ICML 2024 · similarity 0.81)
- Incorporating Surrogate Gradient Norm to Improve Offline Optimization Techniques (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
