Research map / Bandits and online learning / Contextual bandits and Thompson sampling
Off-policy estimation with adaptively collected data: the power of online learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives non-asymptotic MSE bounds for AIPW off-policy/treatment-effect estimators under adaptive data collection and reduces the choice of outcome-model sequence to online non-parametric regression, with a local minimax lower bound showing near-optimality.
Problem
Estimating linear functionals of the treatment effect (off-policy value, ATE) from adaptively collected contextual bandit data, where prior theory is asymptotic and i.i.d.-based cross-fitting fails.
Approach
Derives an MSE upper bound for AIPW estimators depending on a sequentially weighted error of the outcome-model estimates, and constructs those estimates sequentially by running no-regret online regression (OGD) on importance-weighted squared losses, yielding an oracle inequality and a matching local minimax lower bound.
Key results
- Oracle inequality: AIPW MSE bounded by (v*^2 + E[Regret]/n + inf approximation error)/n, showing efficiency when regret is o(n).
- OGD with tabular and linear outcome models attains regret bounds of order 6LB^2 diam(F) sqrt(n) and 6B^2R(L+R) sqrt(n), respectively.
- Local minimax lower bound of order C(K) v*^2 / n, establishing instance-dependent near-optimality of AIPW with no-regret online learners.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sequential treatment-effect estimates in place of the oracle treatment effect | AIPW for off-policy value estimation with adaptively collected data · Off-policy value, adaptively collected data | MSE | Helped |
| Use online regression estimates with regret against a function class | AIPW for off-policy value estimation with adaptively collected data · Off-policy value, adaptively collected data | MSE | Helped |
| Use no-regret online learning to construct treatment-effect estimates | AIPW for off-policy value estimation with adaptively collected data · Off-policy value, adaptively collected data | local minimax risk | Helped |
Limitations
- Requires strict overlap assumption (bounded likelihood ratios) for the regret and bounds.
- Efficiency loss depends on approximation error of the function class for the treatment effect.
- Lower bound relies on existence of reference Markov policies with bounded coverage ratio K.
Experimental setup
Metrics: mean-squared error regret
Baselines: cross-fitted AIPW (i.i.d.) inverse propensity weighting
Builds on: doubly robust estimation cross-fitting online non-parametric regression
Closest papers by meaning
- Optimistic Algorithms for Adaptive Estimation of the Average Treatment Effect (ICML 2025 · similarity 0.91)
- Log-Sum-Exponential Estimator for Off-Policy Evaluation and Learning (ICML 2025 · similarity 0.88)
- Information-Directed Offline-to-Online Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Policy Learning for Balancing Short-Term and Long-Term Rewards (ICML 2024 · similarity 0.87)
- Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning (NeurIPS 2024 · similarity 0.87)
- A General Framework for Off-Policy Learning with Partially-Observed Reward (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.