Research map / Bandits and online learning / Multi-armed bandits
Bandit formulations for ML problems
Cast LLM decoding, sampling, and experimental design as bandit problems.
42 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Multi-armed bandits, keyword cluster “multi armed · armed bandit · stochastic”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
10 papers in 2023–24 and 9 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 9.4% to 7.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Optimistic Gittins Indices (NeurIPS 2016 · 14 citations)
- 2017 · Automated Curriculum Learning for Neural Networks (ICML 2017 · 37 citations)
- 2018 · A Bandit Approach to Sequential Experimental Design with False Discovery Control (NeurIPS 2018 · 11 citations)
- 2019 · Bootstrapping Upper Confidence Bound (NeurIPS 2019 · 24 citations)
- 2020 · BanditPAM: Almost Linear Time k-Medoids Clustering via Multi-Armed Bandits (NeurIPS 2020 · 9 citations)
- 2021 · Multi-layered Network Exploration via Random Walks: From Offline Optimization to Online Learning (ICML 2021 · 4 citations)
- 2022 · Machine Learning for Online Algorithm Selection under Censored Feedback (AAAI 2022 · 6 citations)
- 2023 · AC-Band: A Combinatorial Bandit-Based Approach to Algorithm Configuration (AAAI 2023 · 2 citations)
- 2024 · Reciprocal Learning (NeurIPS 2024 · 2 citations)
- 2025 · Offline Learning for Combinatorial Multi-armed Bandits (ICML 2025 · 0 citations)
- 2026 · KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Bandits (ICML 2026 · 0 citations)
Earliest papers here
- Optimistic Gittins Indices (NeurIPS 2016 · 14 citations)
- Scalable Discrete Sampling as a Multi-Armed Bandit Problem (ICML 2016 · 3 citations)
- Automated Curriculum Learning for Neural Networks (ICML 2017 · 37 citations)
What has been tried
38 papers here report 320 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 7%, hurt 18%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use E-TS instead of random corruption | FashionMNIST, varying number of corrupted clients · ASR | Helped | Constructing Adversarial Examples for Vertical Federated Learning: Optimal Client Corruption through Multi-Armed Bandit (ICLR 2024 · 1 citation) | |
| Use the empirical mean for arrival probabilities instead of a UCB | LLM cache · suboptimality gap bound | a factor of sqrt{m} | Helped | Offline Learning for Combinatorial Multi-armed Bandits (ICML 2025 · 0 citations) |
| Use gradient-based features instead of DINO embeddings for submodular selection | Fashion-MNIST across subset sizes · test accuracy | Helped | Bandit Guided Submodular Curriculum for Adaptive Subset Selection (NeurIPS 2025 · 0 citations) | |
| Increase MDN's initial labeled pool size | VOC2012 · minimum and mean mAP@50 | 1,200 initially labeled samples (40% of budget) | Helped | Adaptive Data Collection for Robust Learning Across Multiple Distributions (ICML 2025) |
What reviewers will expect
Datasets and baselines named in the 6 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR10 · 2 of 6 papers
- MNIST · 2 of 6 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 19 papers; from OpenAlex and the papers' affiliations).
- Columbia University · 3 papers
- Inria · 2 papers
- Microsoft · 2 papers
- Carnegie Mellon University · 2 papers
Use it
Download BibTeX of 32 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Non-stationary and structured-reward bandits (140 papers)
- Optimal stochastic bandit algorithms (124 papers)
- Multi-agent bandits (42 papers)
Elsewhere in the topic
- Best-arm identification in linear bandits (57 papers)
- Pure exploration algorithms (57 papers)
- Fixed-confidence versus fixed-budget identification (35 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).