Research map / Reinforcement learning / Offline RL
Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, Sergey Levine
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Calibrate CQL Q-values against a reference policy | CQL offline pre-training and online fine-tuning · 11-task benchmark | normalized score after fine-tuning: 42 → 71 (CQL average); 44 → 90 (Cal-QL average) | Improved |
| Calibrate CQL Q-values against a reference policy | CQL offline pre-training and online fine-tuning · 11-task benchmark | cumulative regret: 0.38 (IQL average) -> 0.22 | Improved |
| Increase update-to-data ratio to 20 (ablation) | Cal-QL · online fine-tuning benchmark | performance | Improved |
| Use calibration with a narrow-coverage offline dataset rather than uncalibrated CQL (ablation) | CQL · AntMaze medium, narrow-coverage dataset | online fine-tuning performance | Improved |
| Add explicit calibration on a high-coverage diverse dataset (ablation) | CQL · AntMaze medium-diverse | online fine-tuning performance | No clear effect |
| Estimate the reference value with a neural network regressed on Monte-Carlo returns instead of using Monte-Carlo returns directly (ablation) | Cal-QL · FrankaKitchen | performance | No clear effect |
| Increase the offline-data mixing ratio during fine-tuning (ablation) | Cal-QL · Adroit door-binary | performance improvement during fine-tuning | Hurt |
| Vary the online fine-tuning temperature (ablation) | IQL · online fine-tuning benchmark | sample efficiency | No clear effect |
| Increase IQL update-to-data ratio to 5 (ablation) | IQL · online fine-tuning benchmark | asymptotic performance | No clear effect |
| Use a more aggressive policy update (ablation) | IQL · online fine-tuning benchmark | policy performance | Hurt |
Later papers that compared with this method
8 later accepted papers report Cal-QL as a baseline in their results tables; 8 of them report a better number on most of the results they share with it.
- From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learning (ICML 2026 · better on 2 of 2 shared results)
- Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning (ICML 2026 · better on 1 of 1 shared result)
- Tackling Heavy-Tailed Q-Value Bias in Offline-to-Online Reinforcement Learning with Laplace-Robust Modeling (ICLR 2026 · better on 1 of 1 shared result)
- State Proficiency-Based Adaptive Fine-Tuning for Offline-to-Online Reinforcement Learning (AAAI 2026 · better on 1 of 1 shared result)
- Online Pre-Training for Offline-to-Online Reinforcement Learning (ICML 2025 · better on 1 of 1 shared result)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · better on 8 of 8 shared results)
- Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning (ICML 2024 · better on 4 of 4 shared results)
- Revisiting the Minimalist Approach to Offline Reinforcement Learning (NeurIPS 2023 · better on 2 of 2 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.93)
- Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.93)
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- On the Complexity of Offline Reinforcement Learning with Q*-Approximation and Partial Coverage (NeurIPS 2026 · similarity 0.91)
- Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
