AI Paper Atlas

Research map / Reinforcement learning / Offline RL

Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning

Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, Sergey Levine

NeurIPS 2023 · Reinforcement learning › Offline RL · approach: Offline-to-online fine-tuning · 20 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Calibrate CQL Q-values against a reference policyCQL offline pre-training and online fine-tuning · 11-task benchmarknormalized score after fine-tuning: 42 → 71 (CQL average); 44 → 90 (Cal-QL average)Improved
Calibrate CQL Q-values against a reference policyCQL offline pre-training and online fine-tuning · 11-task benchmarkcumulative regret: 0.38 (IQL average) -> 0.22Improved
Increase update-to-data ratio to 20 (ablation)Cal-QL · online fine-tuning benchmarkperformanceImproved
Use calibration with a narrow-coverage offline dataset rather than uncalibrated CQL (ablation)CQL · AntMaze medium, narrow-coverage datasetonline fine-tuning performanceImproved
Add explicit calibration on a high-coverage diverse dataset (ablation)CQL · AntMaze medium-diverseonline fine-tuning performanceNo clear effect
Estimate the reference value with a neural network regressed on Monte-Carlo returns instead of using Monte-Carlo returns directly (ablation)Cal-QL · FrankaKitchenperformanceNo clear effect
Increase the offline-data mixing ratio during fine-tuning (ablation)Cal-QL · Adroit door-binaryperformance improvement during fine-tuningHurt
Vary the online fine-tuning temperature (ablation)IQL · online fine-tuning benchmarksample efficiencyNo clear effect
Increase IQL update-to-data ratio to 5 (ablation)IQL · online fine-tuning benchmarkasymptotic performanceNo clear effect
Use a more aggressive policy update (ablation)IQL · online fine-tuning benchmarkpolicy performanceHurt

Later papers that compared with this method

8 later accepted papers report Cal-QL as a baseline in their results tables; 8 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy