Research map / Bandits and online learning / Contextual bandits and Thompson sampling
Generator-Mediated Bandits: Thompson Sampling for GenAI-Powered Adaptive Interventions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GAMBITTS, Thompson-sampling bandits for settings where an action is a query to a stochastic generator (e.g., an LLM) whose output (treatment) drives the reward, with regret bounds and simulations calibrated to an mHealth study.
Problem
Bandit learning when the agent's action only affects rewards through a stochastic, high-dimensional generated treatment (e.g., LLM-generated text), which standard bandits ignore.
Approach
Model the two stages explicitly (treatment model Z|A,X and reward model E[Y|Z,X]) and run Thompson sampling over the induced expected reward per action, using the observed treatments to estimate rewards; variants include fully online, partially online with offline simulated treatment distributions, and ensemble-based nonlinear reward models.
Key results
- In a single-dimension (optimism) simulation, GAMBITTS variants achieve lower cumulative regret than standard Thompson sampling, with poGAMBITTS performing especially well.
- The linear poGAMBITTS regret is stable as the number of arms K varies (3 to 40), while ens-poGAMBITTS increases only slightly with K.
- Regret bounds: poGAMBITTS with linear reward gives O~(sigma_2 d sqrt(T) + T sqrt(eps T)); foGAMBITTS gives O~(sigma_1 sqrt(CKT) + sigma_2 d sqrt(T)), sharper than standard TS when d << sqrt(CK).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GAMBITTS instead of standard Thompson sampling | Single-dimension optimism reward simulation · Cumulative regret | regret | Helped |
| Use poGAMBITTS instead of standard Thompson sampling | Single-dimension optimism reward simulation · Cumulative regret | regret | Helped |
| Use a treatment embedding correlated with the true reward-relevant dimension (ablation) | GAMBITTS · Treatment embedding misspecification | regret | Helped |
| Decrease correlation between the working embedding and the true reward-relevant dimension (ablation) | GAMBITTS · Treatment embedding misspecification | regret | Hurt |
| Use a low-correlation, reward-misaligned treatment embedding (ablation) | GAMBITTS · Treatment embedding misspecification | regret | Hurt |
| Increase the number of arms (ablation) | Linear poGAMBITTS · Action-space scaling (K=3, K=5, K=15, K=30, K=40) | regret | No clear effect |
| Increase the number of arms (ablation) | ens-poGAMBITTS · Action-space scaling (K=3, K=5, K=15, K=30, K=40) | regret | Hurt |
| Use foGAMBITTS instead of the other methods | Action-space scaling simulation · More complex reward structure | regret | Hurt |
| Pretrain the treatment model using offline generator simulations | GAMBITTS · Online adaptation | performance | Helped |
| Model the stochastic treatment-generation process rather than ignore it | GAMBITTS versus standard Thompson sampling · Bayesian regret | regret bound | Helped |
Surprising findings
- Under low embedding correlation with the true reward-relevant dimension, GAMBITTS can underperform standard Thompson sampling.
Limitations
- Theory assumes finite context space and correctly specified embedding (Z = Z*) in main results.
- Performance degrades when the working treatment embedding is poorly correlated with the true reward-relevant dimension.
- Evaluation is purely simulation-based; no real-world deployment dataset exists for this setting.
Experimental setup
Datasets: 2023 Intern Health Study (IHS) empirical distributions (calibration)
Benchmarks: simulation study calibrated to 2023 Intern Health Study
Metrics: cumulative regret
Baselines: standard Thompson sampling (StdTS) contextual standard Thompson sampling (StdTS:Contextual)
Builds on: Thompson sampling ensemble sampling (Lu and Van Roy 2017) instrument-armed bandit (Kallus 2018) Lattimore and Szepesvári bandit theory
Closest papers by meaning
- RoME: A Robust Mixed-Effects Bandit Algorithm for Optimizing Mobile Health Interventions (NeurIPS 2024 · similarity 0.86)
- Contextual Thompson Sampling via Generation of Missing Data (NeurIPS 2025 · similarity 0.86)
- EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration (ICML 2025 · similarity 0.85)
- A Classification View on Meta Learning Bandits (ICML 2025 · similarity 0.84)
- PRIORITY2REWARD: Incorporating Healthworker Preferences for Resource Allocation Planning (AAAI 2025 · similarity 0.84)
- Towards Cost-Effective Reward Guided Text Generation (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.