Research map / Bandits and online learning / Contextual bandits and Thompson sampling
Only Pay for What Is Uncertain: Variance-Adaptive Thompson Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VarTS, a Gaussian-Gamma Thompson sampling algorithm for Gaussian bandits with unknown heterogeneous reward variances, with a prior-dependent Bayes regret bound and experiments on Bernoulli, beta, and Gaussian bandits.
Problem
Multi-armed bandits where reward variances are unknown and heterogeneous across arms; prior frequentist variance-adaptive algorithms cannot exploit informative priors on variances, and Bayesian analyses have not handled unknown variances.
Approach
Maintain a joint Gaussian-Gamma posterior over each arm's mean and precision, sample precision then mean, and pull the arm with the highest sampled mean; prove a Bayes regret bound that decreases with lower reward variances and more informative variance priors.
Key results
- VarTS matches Bernoulli TS, the best method, in the Bernoulli bandit with K=10 and outperforms the frequentist UCB1 and UCB-V, which have the highest regret.
- For K=32, VarTS has at least 3x lower regret than any baseline in the Bernoulli bandit, at least 5x in the beta bandit, and at least 2x in the Gaussian bandit.
- In the Gaussian bandit, UCB1-Tuned has the lowest regret and VarTS performs similarly, while all other algorithms have at least 50% higher regret.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt to unknown reward variances and use the estimated or true prior | VarTS · Bernoulli bandit (K = 10) | regret | Helped |
| Adapt to unknown reward variances and use the estimated or true prior | VarTS · Beta bandit | regret | Helped |
| Learn reward variances using Thompson sampling | TS14 and TS20 · Beta bandit | regret | Helped |
| Use VarTS instead of UCB1-Tuned | Gaussian bandit · Gaussian bandit | regret | No clear effect |
| Adapt to reward variances and use priors on mean reward and variance | VarTS · Bernoulli bandit (K = 32) | regret: at least 3 times lower regret | Helped |
| Adapt to reward variances and use priors on mean reward and variance | VarTS · Beta bandit (K = 32) | regret: at least 5 times lower regret | Helped |
| Adapt to reward variances and use priors on mean reward and variance | VarTS · Gaussian bandit (K = 32) | regret: at least 2 times lower regret | Helped |
Surprising findings
- VarTS is competitive with Bernoulli TS even though VarTS assumes Gaussian rewards and Bernoulli TS is designed for the Bernoulli problem class.
- Frequentist variance-adaptive UCB algorithms (UCB1, UCB-V) are overly conservative and among the worst performers.
Limitations
- Analysis assumes Gaussian rewards, while experiments include Bernoulli and beta rewards, so theory is not fully matched to experiments.
- Regret bound is not shown to be tight; a matching finite-time lower bound is left open.
- Only K-armed (non-contextual) bandits are considered.
Experimental setup
Benchmarks: Bernoulli bandit (K=10) beta bandit (s=10) Gaussian bandit scalability over K up to 32
Metrics: Bayes regret
Baselines: UCB1 UCB1-Tuned UCB-V Bernoulli TS Gaussian TS TS14 TS20
Builds on: Thompson sampling UCB-V UCB1-Normal TS14 TS20
Closest papers by meaning
- The Choice of Noninformative Priors for Thompson Sampling in Multiparameter Bandit Models (AAAI 2024 · similarity 0.92)
- Noise-Adaptive Confidence Sets for Linear Bandits and Application to Bayesian Optimization (ICML 2024 · similarity 0.91)
- Variance-Aware Feel-Good Thompson Sampling for Contextual Bandits (NeurIPS 2025 · similarity 0.91)
- Improved Regret Analysis in Gaussian Process Bandits: Optimality for Noiseless Reward, RKHS norm, and Non-Stationary Variance (ICML 2025 · similarity 0.91)
- How Does Variance Shape the Regret in Contextual Bandits? (NeurIPS 2024 · similarity 0.90)
- Feel-Good Thompson Sampling for Contextual Bandits: a Markov Chain Monte Carlo Showdown (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.