Research map / Reinforcement learning / Offline RL
Bayesian Design Principles for Offline-to-Online Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BOORL, a Bayesian probability-matching offline-to-online RL method that uses bootstrapped offline ensembles of TD3+BC policies and softmax-Q posterior sampling during online fine-tuning, with information-theoretic regret bounds in linear MDPs.
Problem
Offline-to-online RL faces a dilemma: pessimistic fine-tuning learns too slowly, while optimistic fine-tuning causes sudden performance drops.
Approach
Instead of acting optimistically or pessimistically, act by probability matching (Thompson-sampling-style) over a bootstrapped posterior of policies, sampling a policy per step with softmax-Q weights and reweighting online/offline data 50/50 during updates.
Key results
- BOORL achieves delta_sum of 698.1 over 0.2M online steps across D4RL tasks, versus 392.0 for Cal-QL, 326.8 for PEX and 146.0 for ODT.
- On Hopper-random, BOORL reaches 75.7 normalized score after fine-tuning versus 10.1 to 30.8 for ODT and 9.3 to 11.9 for Cal-QL style baselines (from the table, BOORL 8.8 to 75.7).
- Bernoulli bandit experiments show Thompson-sampling-style Bayesian agent matches LCB early and UCB later, while naive LCB-to-UCB switching yields sudden regret increases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BOORL instead of the compared fine-tuning baselines | Offline-to-online RL · D4RL tasks, 0.2M online steps | δ_sum: BOORL 698.1; Cal-QL 392.0; PEX 326.8; ODT 146.0 | Helped |
| Remove online access to offline data during online fine-tuning (Bayesian) (ablation) | BOORL · Hopper random | normalized score: 75.7±1.3 vs 85.4±3.3 | Mixed |
| Remove online access to offline data during online fine-tuning (Bayesian) (ablation) | BOORL · Hopper medium | normalized score: 109.8±1.6 vs 109.6±1.5 | Hurt |
| Remove online access to offline data during online fine-tuning (Bayesian) (ablation) | BOORL · Walker2d random | normalized score: 93.6±4.4 vs 92.4±4.7 | Hurt |
| Remove online access to offline data during online fine-tuning (Bayesian) (ablation) | BOORL · Walker2d medium | normalized score: 107.7±0.5 vs 96.5±3.5 | Hurt |
| Remove online access to offline data during online fine-tuning (Bayesian) (ablation) | BOORL · Walker2d medium-replay | normalized score: 114.4±0.9 vs 103.7±2.1 | Hurt |
| Remove online access to offline data during online fine-tuning (Bayesian) (ablation) | BOORL · Halfcheetah random | normalized score: 97.7±1.1 vs 94.5±4.2 | Hurt |
| Remove online access to offline data during online fine-tuning (Bayesian) (ablation) | BOORL · Halfcheetah medium | normalized score: 98.7±0.3 vs 97.7±0.5 | Hurt |
| Replace BOORL with Hybrid RL (TD3+BC → TD3) (ablation) | BOORL · Walker2d random | normalized score: 93.6±4.4 vs 15.4±0.8 | Hurt |
| Replace BOORL with Hybrid RL (TD3+BC → TD3) (ablation) | BOORL · Halfcheetah medium | normalized score: 98.7±0.3 vs 80.3±0.2 | Hurt |
| Use a uniform offline/online replay buffer (ablation) | BOORL · D4RL fine-tuning tasks | normalized score | Hurt |
| Set ensemble number to 1 (ablation) | BOORL · D4RL fine-tuning tasks | normalized score | Hurt |
Surprising findings
- Naively interpolating from pessimism to optimism performs poorly regardless of the schedule, supporting probability matching over gradual optimism.
- Theory claims online exploration benefits remain even when offline data has full coverage, in the Bayesian setting.
Limitations
- Theory relies on linear MDP assumptions and informal bounds with idealized posterior, while practice uses approximate bootstrapped ensembles.
- Experiments are limited to D4RL Mujoco and Antmaze with a fixed 0.2M-step online budget in main results.
Experimental setup
Datasets: D4RL Mujoco datasets D4RL Antmaze datasets
Benchmarks: D4RL Mujoco (Hopper, Walker2d, Halfcheetah) D4RL Antmaze
Metrics: normalized score delta_sum improvement over 0.2M steps
Baselines: TD3+BC (online) TD3 (finetune) ODT AWAC PEX Cal-QL RLPD Hybrid RL
Builds on: TD3+BC TD3 Thompson sampling information ratio Cal-QL
Closest papers by meaning
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.93)
- Information-Directed Offline-to-Online Reinforcement Learning (NeurIPS 2026 · similarity 0.92)
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.92)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.