Research map / Reinforcement learning / Offline RL
Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a general offline-to-online RL framework (OCR-CFT) that re-evaluates the offline critic optimistically, aligns it with the offline actor via value alignment, and fine-tunes online under a divergence constraint, applied to SAC, TD3 and PPO.
Problem
Offline-pretrained policies fine-tuned online often suffer sharp initial performance drops due to evaluation and improvement mismatches between offline and online settings; existing O2O methods are tied to specific offline algorithms.
Approach
Before online fine-tuning, re-evaluate the offline policy with fitted Q-evaluation to get optimistic Q-values, then calibrate (align) the critic to the offline actor by anchoring to the Q-value of the policy's most likely action, and finally constrain online updates toward the best historical policy via a Lagrangian divergence penalty.
Key results
- On AntMaze (200k interactions), total normalized score rises from 133.7 to 184.3 for O2PPO versus 116.6 for IQL after fine-tuning (IQL 137.4 to 116.6, O2TD3 128.6 to 142.0, O2SAC 131.2 to 148.0 on the same table).
- On D4RL MuJoCo, O2SAC reaches total 1230.41 at 100k online steps, versus 1204.5 for Off2On and 1099.73 for Cal-QL, with the largest total improvement (+225.54).
- Ablation and Table of initial policies show pi_on after re-evaluation and alignment stays close to the offline policy, sometimes exceeding it (e.g., hopper-medium-replay O2SAC 88.97 to 99.64).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use O2SAC instead of the compared methods | D4RL MuJoCo locomotion · D4RL MuJoCo locomotion total | normalized score: 1230.41(225.54) vs 1204.5(182.93) | Helped |
| use O2PPO rather than IQL | AntMaze · AntMaze total (U-v2, U-D-v2) | normalized score: 133.7→184.3 vs 56.6→35.8 | Helped |
| add policy re-evaluation and value alignment to O2SAC (ablation) | O2SAC from CQL · hopper-expert-v2 online fine-tuning | normalized score | Helped |
| add policy re-evaluation and value alignment rather than using constrained fine-tuning alone (ablation) | O2SAC from CQL · O2SAC online fine-tuning | normalized score | Helped |
| add value alignment (ablation) | O2TD3 from TD3+BC · O2TD3 online fine-tuning | normalized score | Helped |
| add constrained fine-tuning (ablation) | O2TD3 from TD3+BC · O2TD3 online fine-tuning | normalized score | Helped |
| add the auxiliary advantage function (ablation) | O2PPO from IQL · O2PPO online fine-tuning | normalized score | Helped |
| use an auxiliary advantage to constrain policy updates (ablation) | O2PPO from IQL · walker2d-medium-replay-v2 | normalized score | Hurt |
| replace the optimal historical reference policy with a fixed-interval recent policy | O2SAC and O2TD3 · D4RL MuJoCo locomotion | normalized score | No clear effect |
| increase the update-to-data ratio to 10 | O2SAC · D4RL MuJoCo locomotion | normalized score | Helped |
Surprising findings
- Off2On, despite a sharp initial performance drop, sometimes achieves better final performance than the proposed methods in some MuJoCo tasks.
- O2PPO gives little or no gain over directly fine-tuning IQL with PPO on walker2d-medium-replay, and the authors note slower improvement due to the auxiliary advantage constraint.
Limitations
- Requires a reference policy selected via online evaluation, which introduces extra testing information; a fixed-interval variant is explored as an alternative.
- Hyperparameters are not provided for AntMaze tasks and results there are not directly compared with many baselines.
- Theoretical guarantee (Corollary) is asymptotic and relies on assumptions; empirical evaluation limited to simulated D4RL tasks.
Experimental setup
Datasets: D4RL MuJoCo (halfcheetah, hopper, walker2d) D4RL AntMaze
Benchmarks: D4RL MuJoCo locomotion D4RL AntMaze
Metrics: normalized D4RL score
Baselines: AWAC IQL PEX Off2On Cal-QL ACA PROTO PROTO+TD3
Builds on: SAC TD3 PPO CQL TD3+BC RCPO
Closest papers by meaning
- Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.92)
- Online Pre-Training for Offline-to-Online Reinforcement Learning (ICML 2025 · similarity 0.92)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.91)
- SUF: Stabilized Unconstrained Fine-Tuning for Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.