Research map / Reinforcement learning / Offline RL
Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
Jongchan Park, Mingyu Park, Donghwan Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a plug-and-play two-stage pretraining that initializes a shared Q-network's backbone via next-state prediction, then trains it with any offline RL objective to improve data efficiency.
Problem
Offline RL policy learning from limited offline data is hard because effective learning usually needs large, diverse datasets to reduce epistemic uncertainty; the paper targets data efficiency across dataset sizes, qualities, and collection strategies.
Approach
A shared Q-network with a backbone feature extractor and two shallow heads: a transition (next-state prediction) head pretrained by supervised MSE, then the backbone is reused under a Q-value head and fine-tuned with any offline RL objective; a projected Bellman/linear-approximation analysis links higher feature rank to lower Q error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add transition-prediction pretraining | AWAC · D4RL Random HalfCheetah | normalized score: 2.6±0.4 -> 51.1±0.9 | Improved |
| add transition-prediction pretraining | CQL · D4RL Medium Hopper | normalized score: 44.2±10.8 -> 90.6±2.2 | Improved |
| add transition-prediction pretraining | IQL · D4RL Medium Walker2d | normalized score: 83.8±1.5 -> 83.6±1.1 | No clear effect |
| add transition-prediction pretraining | TD3+BC · D4RL Medium Replay Walker2d | normalized score: 58.0±3.6 -> 92.0±1.6 | Improved |
| add transition-prediction pretraining | CQL · D4RL Medium Expert Hopper | normalized score: 111.4±1.2 -> 111.1±0.8 | No clear effect |
| add transition-prediction pretraining | AWAC · D4RL Expert HalfCheetah | normalized score: 94.4±0.8 -> 93.5±0.1 | Hurt |
| add transition-prediction pretraining | DrQ+BC · V-D4RL Medium Cheetah Run | average episode return: 340.33±7.55 -> 379.80±45.83 | Improved |
| add transition-prediction pretraining | DrQ+BC · V-D4RL Medium Replay Walker walk | average episode return: 30.09±0.75 -> 28.68±2.29 | Hurt |
| add transition-prediction pretraining | TD3+BC · D4RL Medium HalfCheetah, 10% data | proxy Q-error: 503.6851 -> 287.7740 | Improved |
| add transition-prediction pretraining | TD3+BC · D4RL Medium Replay Hopper, 10% data | proxy Q-error: 320.1790 -> 319.9531 | No clear effect |
| add transition-prediction pretraining | AWAC · Adroit Human Door | normalized score: 60.82±12.38 -> 29.96±22.43 | Hurt |
| freeze the pretrained backbone during RL training (ablation) | TD3+BC + Ours · D4RL Random Hopper | normalized score: 11.0±0.1 -> 11.59±10.56 | No clear effect |
Experimental setup
Datasets: D4RL Robomimic V-D4RL ExoRL Adroit
Benchmarks: D4RL Robomimic V-D4RL ExoRL Adroit
Metrics: normalized score success rate episode return proxy Q-error latent feature rank
Baselines: AWAC CQL IQL TD3+BC IRIS BCQ DrQ+BC MOPO MOBILE ACL TD3
Builds on: TD3+BC AWAC CQL IQL DrQ+BC
Closest papers by meaning
- Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble (NeurIPS 2021 · similarity 0.92)
- Offline Reinforcement Learning with Implicit Q-Learning (ICLR 2022 · similarity 0.92)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.91)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
