Research map / Learning theory and algorithms / Prediction and decision losses
Adaptive testing and experiments
Sequential decision and testing procedures for validity, treatment effects and instance selection.
201 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Prediction and decision losses, keyword cluster “selection · robust · test”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
52 papers in 2023–24 and 54 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 9.7% to 9.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Scientific Ranking over Heterogeneous Academic Hypernetwork (AAAI 2016 · 37 citations)
- 2017 · Improving the Expected Improvement Algorithm (NeurIPS 2017 · 65 citations)
- 2018 · How Many Properties Do We Need for Gradual Argumentation? (AAAI 2018 · 56 citations)
- 2019 · Subtask Gated Networks for Non-Intrusive Load Monitoring (AAAI 2019 · 142 citations)
- 2020 · Designing Network Design Spaces (CVPR 2020 · 1,958 citations)
- 2021 · Efficient Optimal Selection for Composited Advertising Creatives with Tree Structure (AAAI 2021 · 7 citations)
- 2022 · PRISM: A Rich Class of Parameterized Submodular Information Measures for Guided Data Subset Selection (AAAI 2022 · 22 citations)
- 2023 · Rethinking Safe Control in the Presence of Self-Seeking Humans (AAAI 2023 · 5 citations)
- 2024 · Topological data analysis on noisy quantum computers (ICLR 2024 · 12 citations)
- 2025 · Adaptive Merchant-Centric Risk Control via Unbiased Decision-Making and Dynamic Optimization in E-Commerce (AAAI 2025 · 3 citations)
- 2026 · Subgroup Discovery with the Cox Model (ICML 2026 · 0 citations)
Earliest papers here
- Scientific Ranking over Heterogeneous Academic Hypernetwork (AAAI 2016 · 37 citations)
- ForecastICU: A Prognostic Decision Support System for Timely Prediction of Intensive Care Unit Admission (ICML 2016 · 27 citations)
- Achieving budget-optimality with adaptive schemes in crowdsourcing (NeurIPS 2016 · 27 citations)
What has been tried
150 papers here report 1,189 measured changes (a component added, removed or swapped, compared with a baseline): helped 62%, no clear effect 10%, hurt 22%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Omit the c²σ²x term from the variance model | 1D simulated experiment · Agreement with simulated variance | Hurt | Resolution Limit of Single-Photon LiDAR (CVPR 2024 · 9 citations) | |
| Use 4 attention heads instead of the hp* setting | α=0.001 · amplification | 16.38 (7.37) -> 18.47 (9.03) | No clear effect | Algorithmic Collective Action in Recommender Systems: Promoting Songs by Reordering Playlists (NeurIPS 2024 · 3 citations) |
| Increase the positive scaling weight from 1 to 50 | South Dakota properties, five-fold validation · AUC | 0.772 -> 0.742 | Hurt | Data-Driven Structural Fire Risk Prediction for City Properties (AAAI 2024 · 2 citations) |
| Add smoothness vector to the scale-vector-only variant | NYUv2 · Δm (%) | 2.46 -> 1.16 | Helped | GO4Align: Group Optimization for Multi-Task Alignment (NeurIPS 2024 · 2 citations) |
| Replace a fixed ε in PAS with adaptive ε updates | Synthetic · Tracking error and excess cumulative return | Helped | Adaptive Passive-Aggressive Framework for Online Regression with Side Information (NeurIPS 2024 · 2 citations) | |
| Increase the degree of reviewer miscalibration | Synthetic Swiss NSF review data · expected utility | Mixed | A Principled Approach to Randomized Selection under Uncertainty: Applications to Peer Review and Grant Funding (NeurIPS 2025 · 1 citation) | |
| set ω1=1/T instead of ω1=1/√T | AR(1), T=500 · Power (%) | 22.2 -> 70.0 | Helped | e-GAI: e-value-based Generalized $\alpha$-Investing for Online False Discovery Rate Control (ICML 2025 · 0 citations) |
| Increase the total budget to T=40000 | Synthetic RCT, T=40000 · normalized certificate | within 0.5% | No clear effect | Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects (ICML 2025 · 0 citations) |
| remove per-key tracking from TRobustEst (use RobustEst) | Uniform query sets · queries answered effectively | two orders of magnitude | Helped | Breaking the Quadratic Barrier: Robust Cardinality Sketches for Adaptive Queries (ICML 2025 · 0 citations) |
| use clustered robust rather than non-robust variance estimation | Online Retail, null · false detection rate | 0.168 ± 0.004 -> 0.044 ± 0.002 | Helped | YEAST: Yet Another Sequential Test (NeurIPS 2025 · 0 citations) |
| Remove top-k re-evaluation and re-evaluate the whole archive | QDax · QD-Score | Hurt | Quality-Diversity with Limited Resources (ICML 2024 · 0 citations) | |
| Use the naive nesting bound instead of the depth-16 extended Huber bound | Permanent estimation · convergence | Helped | Estimating the Permanent by Nesting Importance Sampling (ICML 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 34 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-100 · 2 of 34 papers
Baselines
- Energy · 3 of 34 papers
- MSP · 2 of 34 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 106 papers; from OpenAlex and the papers' affiliations).
- Carnegie Mellon University · 7 papers
- University of California, Los Angeles · 5 papers
- University of Cambridge · 5 papers
- Wuhan University · 5 papers
- University of California, Berkeley · 5 papers
- Google · 4 papers
- Amazon · 3 papers
- Cornell University · 3 papers
- Tsinghua University · 3 papers
- University of Science and Technology of China · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Active learning and subset selection (199 papers)
- Distributionally robust optimization (184 papers)
- Data attribution and recourse (160 papers)
- Theory of neural network sensitivity (143 papers)
- Determinantal point process inference (122 papers)
Elsewhere in the topic
- Ranking models from comparisons (127 papers)
- Conformal prediction sets (116 papers)
- Provable post-hoc explanations (92 papers)
- Tree ensemble optimization (77 papers)
- Optimal decision tree learning (75 papers)
- Consistency of surrogate losses (72 papers)
- Losses for decision-making (68 papers)
- Shapley value attribution (63 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).