Research map / Probabilistic and scientific modeling / Bayesian optimization
Context in Public Health for Underserved Communities: A Bayesian Approach to Online Restless Bandits
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BCoR, a hierarchical Bayesian Thompson-sampling method with Whittle index policy for online contextual, non-stationary restless bandits, evaluated on simulations and a data-driven ARMMAN maternal health simulator.
Problem
Allocating scarce interventions to improve adherence in public health programs when RMAB transition dynamics are unknown, horizons are short, and budgets are small relative to the number of arms; existing online RMAB RL methods ignore context and non-stationarity.
Approach
Model arm transition probabilities with a hierarchical Bayesian probit model that shares information within arms (passive to active transitions), across arms (covariates with a shared mean), and over time (spline time effects), then run Thompson sampling by plugging posterior draws into a Whittle index policy.
Key results
- In the ARMMAN data-driven simulator at B=10, BCoR achieves a 61% increase in engagement over the next-best method.
- In a partially misspecified stationary simulation (Figure 1c), BCoR's final time-averaged reward is more than double that of the next-best method, TS.
- In the fully misspecified setting with no structure, none of the methods significantly outperform random over the horizon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BCoR instead of the next-best method | ARMMAN data-driven simulator · ARMMAN data-driven simulator, B=10 | engagement: 61% increase | Helped |
| Use BCoR instead of the strongest competing approach | RMAB simulation with informative covariates and partial misspecification · Figure (c) | time-averaged reward: more than double | Helped |
| Use BCoR instead of existing approaches | RMAB simulations with covariate structure · Figures (a–c) | time-averaged reward | Helped |
| Use BCoR instead of other approaches | RMAB simulation with non-stationarity and uninformative covariates · Figure (d) | time-averaged reward | Helped |
| Use any compared learning method instead of random selection | RMAB simulation with no information sharing or stationarity · Figure (e) | time-averaged reward | No clear effect |
| Use Greedy instead of Whittle policy | BCoR and TS · Simulation experiments | time-averaged reward | No clear effect |
| Use any of the three Whittle oracle variants | Whittle index oracle · Simulation and ARMMAN data-driven experiments | time-averaged reward | No clear effect |
Surprising findings
- Existing online RMAB methods with strong theoretical regret guarantees can perform close to or no better than random selection in moderate sample regimes.
- In the fully unstructured setting, no method including BCoR significantly outperforms random.
Limitations
- Real ARMMAN data and simulator cannot be released due to privacy agreements.
- Linear covariate model and misspecified time model; theoretical regret guarantees not provided.
- Posterior computation via HMC takes up to ~10 minutes per update with N=400, which is slower than simpler Bayesian approaches.
Experimental setup
Datasets: ARMMAN anonymized covariate data (24,011 beneficiaries, 2022) simulated RMAB instances
Benchmarks: simulated RMAB settings (N=400, T=50) ARMMAN data-driven simulator (N=500, T=40)
Metrics: time-averaged reward percent engagement increase over next-best method
Baselines: UCWhittle Thompson sampling per-arm (TS) Random Whittle oracle Greedy oracle
Builds on: Whittle index policy Thompson sampling UCWhittle
Closest papers by meaning
- A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health (NeurIPS 2024 · similarity 0.88)
- PRIORITY2REWARD: Incorporating Healthworker Preferences for Resource Allocation Planning (AAAI 2025 · similarity 0.86)
- RoME: A Robust Mixed-Effects Bandit Algorithm for Optimizing Mobile Health Interventions (NeurIPS 2024 · similarity 0.86)
- Improving Health Information Access in the World’s Largest Maternal Mobile Health Program via Bandit Algorithms (AAAI 2024 · similarity 0.86)
- EpiCare: A Reinforcement Learning Benchmark for Dynamic Treatment Regimes (NeurIPS 2024 · similarity 0.85)
- Using Adaptive Bandit Experiments to Increase and Investigate Engagement in Mental Health (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.