Research map / Reinforcement learning / Offline RL
Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WSRL, which fine-tunes offline RL initializations online without retaining offline data by warming up the replay buffer with a few rollouts from the frozen pre-trained policy and then running high-UTD SAC.
Problem
Online RL fine-tuning of offline RL pre-trained policies and Q-functions typically requires retaining offline data for stability, which is expensive and undermines the benefit of pre-training; existing methods unlearn and forget the pre-trained initialization without it.
Approach
Seed the online replay buffer with a small warmup set (5k steps) of rollouts from the frozen pre-trained policy so Q-values can recalibrate on in-distribution data, then run a standard non-pessimistic high-UTD SAC with ensemble Q-functions and layer norm, initialized from the pre-trained policy and Q-function.
Key results
- WSRL significantly outperforms baselines in seven no-retention fine-tuning tasks, while CQL, IQL, and CalQL fail completely in this setting.
- On the real Franka peg insertion task, WSRL goes from 13/20 offline to 20/20 in 18 minutes, while SERL/RLPD achieves 0/20 after about 50 minutes.
- Removing the 5k-step warmup phase degrades WSRL fine-tuning performance or increases variance across three environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WSRL instead of SERL for online fine-tuning | CalQL-pretrained Franka Peg Insertion · Franka Peg Insertion | successes (20 trials): 0/20 -> 20/20 | Helped |
| Fine-tune with WSRL rather than retain the offline Cal-QL checkpoint | Franka Peg Insertion · Franka Peg Insertion | successes (20 trials): 13/20 -> 20/20 | Helped |
| Add a warmup phase using rollouts from the frozen pretrained policy (ablation) | WSRL · Three environments | fine-tuning performance | Helped |
| Seed the replay buffer with random actions instead of pretrained-policy actions (ablation) | WSRL · Kitchen-mixed and Kitchen-partial | fine-tuning performance | Hurt |
| Use a short warmup phase of 1k steps rather than the tested longer warmups (ablation) | WSRL · Kitchen-mixed, Antmaze-large-diverse, and Door-binary | fine-tuning performance | Mixed |
| Use a longer warmup phase, such as 20k steps (ablation) | WSRL · Kitchen-mixed | fine-tuning performance | Hurt |
| Use CalQL instead of SAC for online fine-tuning (ablation) | WSRL with matched initialization and warmup · Fine-tuning environments | fine-tuning performance | Hurt |
| Remove pretrained policy initialization (ablation) | WSRL · Fine-tuning environments | fine-tuning speed | Hurt |
| Remove pretrained Q-function initialization (ablation) | WSRL · Antmaze | fine-tuning speed | Hurt |
| Remove pretrained Q-function initialization (ablation) | WSRL · Some domains | fine-tuning performance | No clear effect |
| Initialize the Q-function from CQL, IQL, or CalQL rather than CalQL alone (ablation) | WSRL · Kitchen-mixed, Kitchen-partial, and Kitchen-complete | fine-tuning performance | No clear effect |
| Initialize the Q-function from CalQL rather than CQL or IQL (ablation) | WSRL · Kitchen-mixed | fine-tuning performance | Helped |
Surprising findings
- Retaining offline data, which is standard practice, hurts asymptotic performance and efficiency of offline-to-online fine-tuning.
- Any no-retention fine-tuning method shows an initial performance dip, which the authors argue is hard to avoid; freezing the policy for 10k or 30k steps did not prevent it.
Limitations
- Some unlearning (initial performance dip) at the start of fine-tuning remains unavoidable, and freezing the policy does not fix it.
- Warmup length is a hyperparameter that was not extensively tuned; too long warmup can slow online improvement on some tasks such as kitchen-mixed.
- Evaluated mostly in single-task, single-environment settings where pre-training and fine-tuning share the same task.
Experimental setup
Datasets: D4RL Antmaze D4RL Kitchen D4RL Adroit D4RL Mujoco locomotion Antmaze-Ultra Franka peg insertion offline dataset (~17K transitions)
Benchmarks: D4RL Antmaze D4RL Kitchen D4RL Adroit D4RL Mujoco locomotion Franka peg insertion
Metrics: success rate normalized return KL divergence of policy and Q-function TD error
Baselines: JSRL CQL IQL CalQL SO2 RLPD SAC (fast) SERL
Builds on: CalQL CQL IQL SAC RLPD JSRL
Closest papers by meaning
- Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · similarity 0.92)
- Online Pre-Training for Offline-to-Online Reinforcement Learning (ICML 2025 · similarity 0.92)
- SUF: Stabilized Unconstrained Fine-Tuning for Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.92)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.