Research map / Reinforcement learning / Offline RL
Online Pre-Training for Offline-to-Online Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OPT, which adds a third 'Online Pre-Training' phase that trains a new value function on offline and early online data before fine-tuning with a scheduled mixture of old and new critics, improving TD3, SPOT, IQL and RLPD in offline-to-online RL.
Problem
Offline pre-trained agents often underperform during online fine-tuning because the value function is inaccurate under distribution shift, and random re-initialization of the critic disrupts early policy learning.
Approach
Keep the offline critic and add a second critic trained during a short Online Pre-Training phase using a meta-adaptation objective (OEMA-style) on both offline and early online buffers; during fine-tuning the policy maximizes a scheduled convex combination of both critics, shifting weight from the offline to the new critic over time.
Key results
- TD3+OPT reaches 939.1 total normalized score on MuJoCo vs 752.4 for TD3 and 860.9 for Off2On (9 environments, 10 seeds).
- SPOT+OPT reaches 565.3 on Antmaze vs 465.7 for SPOT and 503.4 for Cal-QL; on Adroit SPOT+OPT reaches 309.1 vs 200.2 for SPOT.
- Applying OPT to IQL improves MuJoCo total from 478.7 to 558.5 and Antmaze from 389.7 to 489.1, and RLPD+OPT reaches 918.7 MuJoCo total vs 887.6 for RLPD off-to-on.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add OPT to TD3 | TD3 · D4RL MuJoCo | normalized score (total): 752.4 -> 939.1 | Helped |
| add OPT to SPOT | SPOT · D4RL Antmaze | normalized score (total): 465.7 -> 565.3 | Helped |
| add OPT to SPOT | SPOT · D4RL Adroit | normalized score (total): 200.2 -> 309.1 | Helped |
| add the new value function Q_on-pt (ablation) | OPT · MuJoCo | normalized score (total): 820.6 -> 933.9 | Helped |
| pre-train the new value function rather than randomly initialize it (ablation) | OPT · MuJoCo | normalized score (total): 843.1 -> 933.9 | Helped |
| train the new value function using both B_off and B_on with meta-adaptation rather than only B_on (ablation) | OPT · MuJoCo | normalized score (total): 882.0 -> 933.9 | Helped |
| retain meta-adaptation when pre-training on B_off and B_on (ablation) | OPT · Antmaze | normalized score (total): 405.0 -> 562.6 | Helped |
| use a gradual κ schedule rather than fixed κ (ablation) | OPT · Antmaze large-play | normalized score: 59.2 -> 78.2 | Helped |
| increase online pre-training samples from 6250 to 25000 (ablation) | OPT · MuJoCo | normalized score (total): 861.3 -> 933.9 | Helped |
| use balanced replay rather than online replay (ablation) | OPT · MuJoCo | normalized score (total): 901.3 -> 933.9 | Helped |
Surprising findings
- Random initialization of the new value function consistently underperformed OPT, and pre-training the new critic on the online buffer alone sometimes performed worse than random initialization in the random dataset.
Limitations
- Gains depend on the offline dataset quality and the kappa schedule, which is hand-chosen per dataset category.
- Evaluated only on D4RL locomotion, navigation, and dexterous manipulation tasks with a fixed 300k online step budget.
Experimental setup
Datasets: D4RL halfcheetah/hopper/walker2d (random, medium, medium-replay) v2 D4RL antmaze v2 D4RL adroit cloned v1
Benchmarks: D4RL MuJoCo D4RL Antmaze D4RL Adroit
Metrics: normalized score total normalized score interquartile mean value estimation bias
Baselines: Off2On OEMA PEX ACA FamO2O Cal-QL RLPD BOORL SO2
Builds on: TD3+BC SPOT OEMA Balanced replay (Off2On) RLPD
Closest papers by meaning
- Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.92)
- Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · similarity 0.92)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- SUF: Stabilized Unconstrained Fine-Tuning for Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.89)
- The Generalization Gap in Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.