Research map / Bandits and online learning / Contextual bandits and Thompson sampling
MultiScale Contextual Bandits for Long Term Objectives
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MultiScale Policy Learning, a hierarchical off-policy contextual bandit framework (MSBL) that uses abundant lower-timescale feedback to form data-dependent priors for learning macro policies on sparse long-term feedback.
Problem
Optimizing AI systems for long-term objectives (e.g., user retention, subscription renewal) is hard because long-term feedback is sparse and occurs at a different timescale than short-term interventions like rankings or chat responses.
Approach
Factorize contexts and policies across timescales so that the lower-level policy family (learned from abundant short-term data) serves as the macro action space, and a macro policy is learned from scarce long-term data; PAC-Bayes analysis motivates the hierarchical prior.
Key results
- On the multi-turn conversation task, MSBL achieves the best long-term (5-turn, LLM-judged) reward overall and per user persona, while single-level policies are optimal for only one user group.
- In the KuaiRand recommender, MSBL maintains a high user return rate across 2-5 user groups compared to baselines, with some sacrifice in short-term clicks.
- In a toy setting, MSBL needs 10 to 62 times fewer macro samples than tabular Q-learning for the same long-term reward as k varies in {2,4,6}.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn a persona-conditioned macro policy for the helpfulness-harmlessness tradeoff | MSBL · Multi-turn Conversation | long-term multi-turn reward | Helped |
| optimize only the per-turn response instead of learning a macro policy | multi-turn conversation policies · Multi-turn Conversation | long-term multi-turn reward | Hurt |
| learn nested contextual policies across two levels | MSBL · Conversational recommender system, two levels | long-term reward | Helped |
| optimize the short-term reward without a macro intervention | Level 1 policy · Conversational recommender system, three levels | long-term reward | Hurt |
| add a third-level policy to the two-level MSBL policy | MSBL · Conversational recommender system, three levels | long-term reward | Helped |
| learn a contextual macro policy that selects ranking boosts | MSBL · KuaiRand recommender system | user return rate | Helped |
| apply no macro intervention and optimize clicks only | Level 1 policy · KuaiRand recommender system | user return rate | Hurt |
| use MSBL rather than Q-learning for learning the long-term policy | MSBL · Toy domain | episodic samples required for the same expected long-term reward: 10 ≤ n_0/n^{L2} ≤ 62 | Helped |
Surprising findings
- Optimizing only the per-turn short-term reward can adversely affect multi-turn conversation outcomes due to responses that are acceptable individually but harmful over five turns.
- A Gaussian PAC-Bayes toy analysis claims ~98% macro sample savings (88.2% when including micro samples).
Limitations
- Lack of public real-world multi-scale datasets; experiments rely on simulators.
- Restricted to contextual bandits; stateful policies are left for future work.
- Assumes i.i.d. contexts.
Experimental setup
Datasets: Anthropic Helpful Assistant KuaiRand
Benchmarks: Anthropic Helpful Assistant simulator simulated conversational recommender KuaiRand-based recommender simulator toy episodic MDP
Metrics: short-term reward long-term reward weekly return rate clicks sample efficiency ratio n0/nL2
Baselines: single-stage short-term policy random policy oracle skyline policy Q-learning
Builds on: PAC-Bayes generalization bounds counterfactual risk minimization off-policy contextual bandits
Closest papers by meaning
- Balancing Context Length and Mixing Times for Reinforcement Learning at Scale (NeurIPS 2024 · similarity 0.87)
- Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF (ICLR 2024 · similarity 0.87)
- Diffusion Models Meet Contextual Bandits (NeurIPS 2025 · similarity 0.87)
- Active Context Selection Improves Simple Regret in Contextual Bandits (NeurIPS 2026 · similarity 0.87)
- Cost-Aware Near-Optimal Policy Learning (AAAI 2025 · similarity 0.86)
- Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.