Research map / Learning theory and algorithms / Prediction and decision losses
Active learning and subset selection
Select informative samples or subsets with optimization-based and importance-weighted algorithms.
199 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Prediction and decision losses, keyword cluster “selection · robust · test”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
57 papers in 2023–24 and 40 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 10.7% to 6.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Geometric Mean Metric Learning (ICML 2016 · 57 citations)
- 2017 · Optimized Pre-Processing for Discrimination Prevention (NeurIPS 2017 · 386 citations)
- 2018 · CatBoost: unbiased boosting with categorical features (NeurIPS 2018 · 1,125 citations)
- 2019 · Efficient Lifelong Learning with A-GEM (ICLR 2019 · 641 citations)
- 2020 · Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds (ICLR 2020 · 274 citations)
- 2021 · Contemporary Symbolic Regression Methods and their Relative Performance (NeurIPS 2021 · 143 citations)
- 2022 · Towards Robust and Reproducible Active Learning Using Neural Networks (CVPR 2022 · 43 citations)
- 2023 · Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities (ICML 2023 · 25 citations)
- 2024 · Benchmark Data Repositories for Better Benchmarking (NeurIPS 2024 · 5 citations)
- 2025 · What Does It Take to Build a Performant Selective Classifier? (NeurIPS 2025 · 2 citations)
- 2026 · Adaptively Robust Resettable Streaming (ICML 2026 · 0 citations)
Earliest papers here
- Geometric Mean Metric Learning (ICML 2016 · 57 citations)
- Higher-Order Factorization Machines (NeurIPS 2016 · 50 citations)
- Nested Mini-Batch K-Means (NeurIPS 2016 · 36 citations)
What has been tried
160 papers here report 1,398 measured changes (a component added, removed or swapped, compared with a baseline): helped 62%, no clear effect 12%, hurt 19%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use the product kernel combining input and learner-output kernels instead of the input kernel alone | FashionMNIST training set, 4,500 labels · mean absolute deviation | 34% | Helped | Active Learning Guided by Efficient Surrogate Learners (AAAI 2024 · 3 citations) |
| Remove offline image-text alignment filtering | ImageNet-1K zero-shot · accuracy | 76.9 vs. 76.7 | No clear effect | Data curation via joint example selection further accelerates multimodal learning (NeurIPS 2024 · 3 citations) |
| Use the full generator-assisted method instead of the V=U variant | Four datasets · oracle-query cost at the same accuracy | 3--5x | Helped | Generator Assisted Mixture of Experts for Feature Acquisition in Batch (AAAI 2024 · 2 citations) |
| Vary dataset size from 0.1 to 1.0 | CIFAR10, alpha=0.1 · Unlearning Quality | No clear effect | A Reliable Cryptographic Framework for Empirical Machine Unlearning Evaluation (NeurIPS 2025 · 1 citation) | |
| Use linear rather than constant auxiliary models | Synthetic regression · R² | 0.814 -> 0.819 | No clear effect | SMLE: Safe Machine Learning via Embedded Overapproximation (AAAI 2025 · 0 citations) |
| Use 5 rather than 4 estimator sampling points | CIFAR-10 · MSE | 0.057668 -> 0.016326 | Helped | Computational Budget Should Be Considered in Data Selection (NeurIPS 2025 · 0 citations) |
| Add the Stage 2 testing procedure after candidate screening | QuALITY · conditional accuracy gap | Helped | Early Time Classification with Accumulated Accuracy Gap Control (ICML 2024 · 0 citations) | |
| Use the efficient selection realization that subtracts orthogonal components from Sum, instead of orthogonalizing all candidate features | Online batch selection · selection time | Helped | Diversified Batch Selection for Training Acceleration (ICML 2024 · 0 citations) | |
| Train on all aggregate responses seen so far rather than only the current slice | Synthetic logistic regression · test log loss | Hurt | PriorBoost: An Adaptive Algorithm for Learning from Aggregate Responses (ICML 2024 · 0 citations) | |
| Increase search times from T=100 to T=500 | F-MNIST, k=1000 and 2000 · test accuracy (%) | 77.0±1.8 -> 79.7±0.7; 80.2±1.9 -> 82.8±0.6 | Helped | Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints (ICML 2024 · 0 citations) |
| Add 76 expert-engineered features to the traditional feature set | Hemisphere datasets, 3 mice · false-positive rejection accuracy | 67% -> 72% | Helped | ActSort: An active-learning accelerated cell sorting algorithm for large-scale calcium imaging datasets (NeurIPS 2024 · 0 citations) |
| Concatenate the time embedding directly to input images instead of to a layer before softmax | CIFAR-10 tasks · prospective risk | worse by almost 0.2 | Hurt | Prospective Learning: Learning for a Dynamic Future (NeurIPS 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 19 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 3 of 19 papers
- CIFAR-100 · 2 of 19 papers
- MNIST · 2 of 19 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 97 papers; from OpenAlex and the papers' affiliations).
- Google · 9 papers
- Stanford University · 4 papers
- University of Cambridge · 4 papers
- Alibaba Group · 3 papers
- Johns Hopkins University · 3 papers
- University of Texas, Austin · 3 papers
- Carnegie Mellon University · 3 papers
- Google DeepMind · 3 papers
- Massachusetts Institute of Technology · 3 papers
- University of California, Berkeley · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Adaptive testing and experiments (201 papers)
- Distributionally robust optimization (184 papers)
- Data attribution and recourse (160 papers)
- Theory of neural network sensitivity (143 papers)
- Determinantal point process inference (122 papers)
Elsewhere in the topic
- Ranking models from comparisons (127 papers)
- Conformal prediction sets (116 papers)
- Provable post-hoc explanations (92 papers)
- Tree ensemble optimization (77 papers)
- Optimal decision tree learning (75 papers)
- Consistency of surrogate losses (72 papers)
- Losses for decision-making (68 papers)
- Shapley value attribution (63 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).