Research map / Reinforcement learning / Visual and representation RL
A Deployed Online Reinforcement Learning Algorithm in an Oral Health Clinical Trial
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Describes the design and post-deployment re-sampling analysis of Oralytics, an online Thompson-sampling contextual bandit RL algorithm that decides when to send toothbrushing prompts in a 79-participant clinical trial.
Problem
Deploying an online RL algorithm inside a registered NIH clinical trial raises challenges of autonomy/replicability (pre-registration, no ad hoc changes) and learning from limited per-participant data (140 decision times), for timing engagement prompts to improve oral self-care behaviors.
Approach
A full-pooling Thompson-sampling contextual bandit (a generalization with a generalized logistic action-selection probability and action-centered linear reward model) that outputs full 70-day action schedules with pre-specified fallback methods, evaluated post hoc via a simulator built from trial data and a re-sampling test of learning.
Key results
- Full pooling achieved mean OSCB 69.724 vs 69.375 for no pooling in the simulation (500 Monte Carlo reps), and first-quartile 43.049 vs 43.024.
- All three fallback methods were triggered during the trial; fallback (i) was executed 4 times and covered more participants than other methods.
- Re-sampling analysis indicates the algorithm learned that prompts are effective in some states (e.g., poor brushing, evening) and ineffective in others, while one state's apparent learning likely arose by chance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use full pooling instead of no pooling (ablation) | Oralytics online RL algorithm · Oralytics simulation environment | Mean OSCB: 69.375 (0.047) -> 69.724 (0.047) | Helped |
| Use full pooling instead of no pooling (ablation) | Oralytics online RL algorithm · Oralytics simulation environment | First quartile OSCB: 43.024 (0.088) -> 43.049 (0.091) | Helped |
| Send an engagement prompt in the evening when recent brushing is poor, recent prompt exposure is low, and the participant did not open the app the prior day | Oralytics online RL algorithm · Evening; about 28 seconds recent OSCB; prompts 20% of the time; app not opened prior day | Predicted advantage of prompting | Helped |
| Send an engagement prompt in the morning when recent brushing is almost ideal, recent prompt exposure is high, and the participant opened the app the prior day | Oralytics online RL algorithm · Morning; about 100 seconds recent OSCB; prompts 45% of the time; app opened prior day | Predicted advantage of prompting | Hurt |
| Interpret the apparent prompting advantage as a real learned effect | Oralytics online RL algorithm · Morning; about 28 seconds recent OSCB; prompts 45% of the time; app not opened prior day | Predicted advantage of prompting | No clear effect |
Surprising findings
- Full pooling beat no pooling only marginally (69.724 vs 69.375), and in the first-quartile metric the difference was negligible (43.049 vs 43.024).
- One of 16 states showed apparent learning that the re-sampling analysis attributes to random chance.
Limitations
- Simulation environment is built from trial data and relies on fitted participant-environment models whose fit quality is approximate (RMSE about 78.5).
- Analysis is post-deployment and based on a single trial with 72 analyzed participants.
- Full pooling assumes participant homogeneity despite acknowledged heterogeneity.
Experimental setup
Datasets: Oralytics clinical trial data (79 enrolled, 72 analyzed) Oralytics pilot study data (for prior)
Benchmarks: Oralytics simulation environment
Metrics: average of participants' average OSCB first quartile of participants' average OSCB standardized predicted advantage
Baselines: no-pooling algorithm
Builds on: Thompson sampling action centering full pooling bandit
Closest papers by meaning
- EpiCare: A Reinforcement Learning Benchmark for Dynamic Treatment Regimes (NeurIPS 2024 · similarity 0.83)
- Context in Public Health for Underserved Communities: A Bayesian Approach to Online Restless Bandits (AAAI 2025 · similarity 0.83)
- RoME: A Robust Mixed-Effects Bandit Algorithm for Optimizing Mobile Health Interventions (NeurIPS 2024 · similarity 0.82)
- Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination (ICML 2024 · similarity 0.81)
- OralGPT-Plus: Learning to Use Visual Tools via Reinforcement Learning for Panoramic X-ray Analysis (CVPR 2026 · similarity 0.81)
- Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies (NeurIPS 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.