Research map / Bandits and online learning / Multi-armed bandits
Efficient Graph Bandit Learning with Side-Observations and Switching Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes graph bandit learning with graph feedback (probabilistic side-observations) and graph switching constraints, giving graph-agnostic UCB-based algorithms (UCB-GG, UCB-GGmax) and a graph-aware LP-based elimination algorithm (LP-GG) with regret bounds and experiments vs Q-learning and LocalUCB.
Problem
Multi-armed bandits where the agent moves on a constraint graph (switching constraints/costs) and receives probabilistic graph-structured side observations from a feedback graph; goal is low regret with few switches while exploiting side observations.
Approach
Combine a UCB-based exploration rule with graph-structured side observations, using offline shortest-path planning (OSSP), mini-batch/doubling-trick epochs to limit switches; a graph-aware variant solves an LP for minimum expected pulls to observe every arm and eliminates suboptimal nodes iteratively.
Key results
- LP-GG outperforms UCB-GG and UCB-GGmax on k-tree graphs, where ||z*||_1 is 1 while independence number is |V|-1, with regret improvement up to roughly sqrt(|V|-1) factor per the paper's claim.
- Local UCB regret grows almost linearly, while Q-learning is sublinear but less stable and needs three tuned hyperparameters.
- Algorithms achieve sub-linear regret on real-world road (California) and Amazon product datasets with |V| = 20 synthetic graph settings and up to 1e6 rounds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the graph-agnostic method rather than generic reinforcement learning methods | Graph bandit with deterministic feedback graph · Graph bandit with deterministic feedback graph | regret order: a constant factor of √|V| | Helped |
| use the graph-agnostic method rather than generic reinforcement learning methods | Graph bandit · Graph bandit | space complexity: O(|V|²) -> O(|V|) | Helped |
| use LP-GG instead of UCB-GG on graphs with highly informative nodes | Graph bandit · k-tree graphs | regret: ∥z∗∥1 is just equal to 1 | Helped |
| use LP-GG instead of UCB-GG on dense graphs | Graph bandit · perfect graphs or clique groups | regret | Hurt |
| use Local UCB, which chooses the highest-UCB node in the current constraint neighborhood | Graph bandit · simulated graph instances | regret | Hurt |
| use Q-learning instead of the proposed graph-structured algorithms | Graph bandit · simulated graph instances | regret | Hurt |
| increase the minimum observation probability p∗ | UCB-GG and UCB-GGmax · random feedback and constraint graphs | regret | Helped |
| increase the maximal switching cost D | UCB-GG and UCB-GGmax · fully connected reward graph and line constraint graph | regret | Hurt |
| use UCB-GGmax instead of UCB-GG | Graph bandit · random feedback and constraint graphs | regret | Helped |
Surprising findings
- LP-GG performs worse than UCB-GG and UCB-GGmax on dense graphs (perfect graphs, clique groups) due to excessive exploration.
- Local UCB, a natural greedy baseline, gets trapped in local reward maxima with near-linear regret.
Limitations
- Regret bounds for graph-aware methods depend on independence number or LP quantities that are NP-hard to compute in general.
- Experiments use synthetic rewards (Gaussian with fixed variance) on mostly small graphs (|V|=20 to 900) and real datasets are synthetically modeled.
- UCB-GG's gap-dependent bound includes a last term that can be large for small T; Algorithm 2 underperforms on dense graphs.
Experimental setup
Datasets: California road network (Leskovec and Krevl 2014) Amazon product co-purchase dataset (Leskovec and Krevl 2014)
Benchmarks: synthetic graph bandit instances (star, k-tree, connected-star, perfect, clique group, random) California road dataset Amazon product dataset
Metrics: cumulative regret
Baselines: Q-learning LocalUCB UCB-GG UCB-GGmax
Builds on: UCB Simchi-Levi and Xu 2019 switching budget bandits Dann et al. 2020 graph bandit RL Mannor and Shamir 2011 feedback graphs Zhang, Johansson, and Li 2022 shortest path planning
Closest papers by meaning
- Reinforcement Learning with Feedback Graphs (NeurIPS 2020 · similarity 0.89)
- Information Directed Sampling for Stochastic Bandits With Graph Feedback (AAAI 2018 · similarity 0.89)
- Bandits with Feedback Graphs and Switching Costs (NeurIPS 2019 · similarity 0.88)
- Stochastic contextual bandits with graph feedback: from independence number to MAS number (NeurIPS 2024 · similarity 0.88)
- Graph-Triggered Rising Bandits (ICML 2024 · similarity 0.87)
- Efficient Contextual Bandits with Uninformed Feedback Graphs (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.