Research map / Bandits and online learning
Multi-armed bandits: research map
537 papers on Multi-armed bandits in Bandits and online learning, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 2 clusters and 7 approaches. The busiest year so far is 2021.
What it is
A bandit problem is choosing repeatedly among options ("arms") with unknown payoffs, as in A/B testing or ad selection. The learner must trade off trying arms to learn about them against playing the one that looks best; pure-exploration variants just aim to identify the best arm quickly.
Terms you will see on this page:
- Arm
- one of the options a bandit algorithm can choose.
- Exploration-exploitation trade-off
- gathering information versus using what is known.
- Best-arm identification
- finding the best arm with as few pulls as possible.
Start reading with the most cited papers here:
- Minimal Exploration in Structured Stochastic Bandits (NeurIPS 2017 · 79 citations)
- Combinatorial Multi-Armed Bandit with General Reward Functions (NeurIPS 2016 · 75 citations)
Within Bandits and online learning, its share grew from 19.0% in 2023–24 to 21.0% in 2025–26 (106 → 125 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Multi-armed bandits in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Optimal stochastic bandit algorithms rising: 20% → 30% of the topic (21 → 37 papers)
- Fixed-confidence versus fixed-budget identification rising: 5% → 14% of the topic (5 → 17 papers)
- Pure exploration algorithms rising: 8% → 13% of the topic (9 → 16 papers)
- Non-stationary and structured-reward bandits falling: 28% → 20% of the topic (30 → 25 papers)
- Multi-agent bandits falling: 16% → 8% of the topic (17 → 10 papers)
Papers from 2025–2026 that are already cited most:
- Quantum Best Arm Identification with Quantum Oracles (AAAI 2025 · 2 citations)
- Nearly Tight Bounds for Exploration in Streaming Multi-Armed Bandits with Known Optimality Gap (AAAI 2025 · 1 citations)
- Balancing Performance and Costs in Best Arm Identification (NeurIPS 2025 · 0 citations)
Approaches and key papers
Stochastic multi-armed bandits · 380 papers
Keywords: multi armed, armed bandit, stochastic
Approaches in this cluster:
- Non-stationary and structured-reward bandits (152 papers)
Handle rising, restless, blocking, and recovering reward dynamics in bandit algorithms. - Optimal stochastic bandit algorithms (136 papers)
Characterize regret-optimal algorithms for stochastic, batched, and combinatorial bandits. - Multi-agent bandits (45 papers)
Coordinate multiple or federated agents in decentralized bandit learning. - Bandit formulations for ML problems (47 papers)
Cast LLM decoding, sampling, and experimental design as bandit problems.
Most cited and most cited since 2024:
- Minimal Exploration in Structured Stochastic Bandits (NeurIPS 2017 · 79 citations)
- Combinatorial Multi-Armed Bandit with General Reward Functions (NeurIPS 2016 · 75 citations)
- Exploration, Exploitation, and Engagement in Multi-Armed Bandits with Abandonment (NeurIPS 2024 · 8 citations)
- Online Restless Multi-Armed Bandits with Long-Term Fairness Constraints (AAAI 2024 · 7 citations)
Best-arm identification · 157 papers
Keywords: identification, best arm, pure exploration
Approaches in this cluster:
- Best-arm identification in linear bandits (61 papers)
Identify the best arm with optimal sample complexity for linear and fixed-budget settings. - Pure exploration algorithms (58 papers)
Design efficient pure-exploration and thresholding algorithms with batch and combinatorial structure. - Fixed-confidence versus fixed-budget identification (38 papers)
Develop optimal elimination and top-two algorithms for confidence and budget settings.
Most cited and most cited since 2024:
- An optimal algorithm for the Thresholding Bandit Problem (ICML 2016 · 64 citations)
- Refined Lower Bounds for Adversarial Bandits (NeurIPS 2016 · 55 citations)
- Thompson Sampling for Real-Valued Combinatorial Pure Exploration of Multi-Armed Bandit (AAAI 2024 · 5 citations)
- Quantum Best Arm Identification with Quantum Oracles (AAAI 2025 · 2 citations)
