Research map / Bandits and online learning / Contextual bandits and Thompson sampling
Practical contextual bandit extensions
Tune, infer from, cluster, or meta-learn contextual bandit algorithms.
77 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Contextual bandits and Thompson sampling, keyword cluster “contextual bandits · contexts · user”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
26 papers in 2023–24 and 23 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 17.4% to 17.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Phased Exploration with Greedy Exploitation in Stochastic Combinatorial Partial Monitoring Games (NeurIPS 2016 · 52 citations)
- 2017 · Interactive Submodular Bandit (NeurIPS 2017 · 14 citations)
- 2018 · Transferable Contextual Bandit for Cross-Domain Recommendation (AAAI 2018 · 73 citations)
- 2019 · Contextual Multi-armed Bandit Algorithm for Semiparametric Reward Model (ICML 2019 · 7 citations)
- 2020 · Contextual-Bandit Based Personalized Recommendation with Time-Varying User Interests (AAAI 2020 · 33 citations)
- 2021 · Statistical Inference with M-Estimators on Adaptively Collected Data (NeurIPS 2021 · 16 citations)
- 2022 · Conformal Off-Policy Prediction in Contextual Bandits (NeurIPS 2022 · 3 citations)
- 2023 · Efficient Explorative Key-Term Selection Strategies for Conversational Contextual Bandits (AAAI 2023 · 10 citations)
- 2024 · Using Adaptive Bandit Experiments to Increase and Investigate Engagement in Mental Health (AAAI 2024 · 11 citations)
- 2025 · Stochastic Online Instrumental Variable Regression: Regrets for Endogeneity and Bandit Feedback (AAAI 2025 · 1 citation)
- 2026 · Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards (ICML 2026 · 1 citation)
Earliest papers here
- Phased Exploration with Greedy Exploitation in Stochastic Combinatorial Partial Monitoring Games (NeurIPS 2016 · 52 citations)
- Interactive Submodular Bandit (NeurIPS 2017 · 14 citations)
- Transferable Contextual Bandit for Cross-Domain Recommendation (AAAI 2018 · 73 citations)
What has been tried
64 papers here report 463 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 9%, hurt 17%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Replace clustering-based dueling bandits with double Thompson sampling | Synthetic test problems · approximation accuracy ε* and average accuracy ε̄ | Hurt | Direct Preference-Based Evolutionary Multi-Objective Optimization with Dueling Bandits (NeurIPS 2024 · 4 citations) | |
| use OFUL-IV instead of one-stage OFUL reduction | Linear bandits with endogeneity · regret | Helped | Stochastic Online Instrumental Variable Regression: Regrets for Endogeneity and Bandit Feedback (AAAI 2025 · 1 citation) | |
| Replace the bagged stochastic gradient tree baseline model with bagged linear models | Nonlinear simulation · cumulative regret | No clear effect | RoME: A Robust Mixed-Effects Bandit Algorithm for Optimizing Mobile Health Interventions (NeurIPS 2024 · 1 citation) | |
| Increase the number of arms K | Square synthetic reward · cumulative regret | Hurt | Neural Dueling Bandits: Preference-Based Optimization with Human Feedback (ICLR 2025 · 0 citations) | |
| use MA-TARI Greedy instead of MA-TARI Integer Linear Programming | Kilkari intervention simulation · additional content listened (hours over control) | 49.6h vs 49.47h | No clear effect | Improving Health Information Access in the World’s Largest Maternal Mobile Health Program via Bandit Algorithms (AAAI 2024 · 0 citations) |
| use CascadingVI instead of CascadingVI-Oracle | MovieLens · regret | No clear effect | Cascading Reinforcement Learning (ICLR 2024 · 0 citations) | |
| Replace the WK kernel with the CK kernel | MovieLens, small arm space, contextual and non-contextual reward settings · cumulative regret | Helped | Gaussian Process Bandits for Top-k Recommendations (NeurIPS 2024 · 0 citations) | |
| Use the Lipschitz reward estimator instead of the binary estimator | MNIST · Average Progressive Reward | 0.614 (0.012) -> 0.638 (0.042) | Helped | Provably Efficient Interactive-Grounded Learning with Personalized Reward (NeurIPS 2024 · 0 citations) |
| Use lambda regularization instead of the doubly robust pseudo-inverse | Random logging policy, correctly specified F and FN, n=400/1000/1600 · average relative regret ratio | 2.47% -> 2.23%; 0.59% -> 0.40%; 0.32% -> 0.18% | Helped | Contextual Linear Optimization with Bandit Feedback (NeurIPS 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 14 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- Thompson sampling · 2 of 14 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 49 papers; from OpenAlex and the papers' affiliations).
- Cornell University · 4 papers
- Google · 3 papers
- Stanford University · 3 papers
- Google DeepMind · 3 papers
- University of Illinois, Urbana Champaign · 3 papers
- The Chinese University of Hong Kong · 2 papers
- University of Wisconsin - Madison · 2 papers
- Harvard University · 2 papers
- Hong Kong University of Science and Technology · 2 papers
- University of Chicago · 2 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Oracle-efficient contextual bandits (168 papers)
- Linear and generalized linear bandits (88 papers)
- Causal and structured bandits (61 papers)
- Ranking bandits (42 papers)
Elsewhere in the topic
- Thompson sampling analysis (84 papers)
- Gaussian process bandit regret (26 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).