Research map / Bandits and online learning / RL theory and MDPs

Information-Directed Offline-to-Online Reinforcement Learning

NeurIPS 2026 · Bandits and online learning › RL theory and MDPs · approach: Linear function approximation in RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formalises offline-to-online residual uncertainty as conditional mutual information and proposes information-directed sampling (IDS) with a regret bound via a ratio certificate, a dominated-probe separation from Thompson sampling, and ROID, a deep IDS selector on TD3+BC ensembles evaluated on D4RL.

Problem

Offline-to-online learning warm-starts a policy from offline data, but the remaining exploration problem (residual uncertainty after conditioning on offline data) is not isolated; existing methods rely on heuristics for the offline-to-online transition.

Approach

Use information-directed sampling, selecting actions that minimise squared regret over information gain conditioned on the offline dataset; a reference Thompson sampling policy serves as a witness so that its information-ratio bound transfers to IDS, and the summed information gains collapse to the conditional mutual information I(chi; tau_{1:T} | D_N).

Key results

• ROID achieves the highest summed online D4RL score of 648.9 versus 633.2 for BOORL across six tasks, best on 4 of 6 tasks.
• On the hidden-mode bandit at N=1000 (p_N=0.0066), IDS0 achieves cumulative regret 0.15 versus 3.31 for greedy/UCB and 1.79 for TS.
• On the biased contextual bandit at N=20, IDS0.5 achieves regret 3.57 versus 10.64 for TS and 42.85 for greedy.

Surprising findings

• Greedy and UCB fail entirely at N=1000 in the hidden-mode bandit, while vanilla IDS probes the dominated action; regularised IDS with eta>0 can fail there.
• In the biased contextual bandit at N=100, UCB outperforms IDS0.5, showing regularisation over-regularises when the warm start is accurate.

Limitations

• Theoretical guarantees hold only for the known-dynamics Bayesian linear-reward model; the deep ROID variant uses ensemble surrogates without formal guarantees.
• D4RL experiments use only six locomotion tasks with 3 seeds for ROID and no code or data release is stated.
• Structural separation is shown on a finite hand-constructed instance, not a universal result.

Experimental setup

Datasets: D4RL hopper-medium-v2 D4RL hopper-medium-replay-v2 D4RL walker2d-medium-v2 D4RL walker2d-medium-replay-v2 D4RL halfcheetah-medium-v2 D4RL halfcheetah-medium-replay-v2

Benchmarks: D4RL hopper, walker2d, halfcheetah (medium, medium-replay) hidden-mode bandit biased linear contextual bandit

Metrics: cumulative regret normalised return

Baselines: greedy UCB TS RLPD ODT PEX Cal-QL BOORL

Builds on: information-directed sampling Thompson sampling information ratio TD3+BC BOORL

Closest papers by meaning

• Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.92)
• Offline-Online Reinforcement Learning for Linear Mixture MDPs (NeurIPS 2026 · similarity 0.91)
• From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
• Sample Complexity of Distributionally Robust Off-Dynamics Reinforcement Learning with Online Interaction (ICML 2025 · similarity 0.90)
• Selective Uncertainty Propagation in Offline RL (AAAI 2025 · similarity 0.90)
• Information-Directed Pessimism for Offline Reinforcement Learning (ICML 2024 · similarity 0.90)