AI Paper Atlas

Research map / Reinforcement learning / Offline RL

Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning

Jongchan Park, Mingyu Park, Donghwan Lee

NeurIPS 2025 · Reinforcement learning › Offline RL · approach: Simple practical offline RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a plug-and-play two-stage pretraining that initializes a shared Q-network's backbone via next-state prediction, then trains it with any offline RL objective to improve data efficiency.

Problem

Offline RL policy learning from limited offline data is hard because effective learning usually needs large, diverse datasets to reduce epistemic uncertainty; the paper targets data efficiency across dataset sizes, qualities, and collection strategies.

Approach

A shared Q-network with a backbone feature extractor and two shallow heads: a transition (next-state prediction) head pretrained by supervised MSE, then the backbone is reused under a Q-value head and fine-tuned with any offline RL objective; a projected Bellman/linear-approximation analysis links higher feature rank to lower Q error.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add transition-prediction pretrainingAWAC · D4RL Random HalfCheetahnormalized score: 2.6±0.4 -> 51.1±0.9Improved
add transition-prediction pretrainingCQL · D4RL Medium Hoppernormalized score: 44.2±10.8 -> 90.6±2.2Improved
add transition-prediction pretrainingIQL · D4RL Medium Walker2dnormalized score: 83.8±1.5 -> 83.6±1.1No clear effect
add transition-prediction pretrainingTD3+BC · D4RL Medium Replay Walker2dnormalized score: 58.0±3.6 -> 92.0±1.6Improved
add transition-prediction pretrainingCQL · D4RL Medium Expert Hoppernormalized score: 111.4±1.2 -> 111.1±0.8No clear effect
add transition-prediction pretrainingAWAC · D4RL Expert HalfCheetahnormalized score: 94.4±0.8 -> 93.5±0.1Hurt
add transition-prediction pretrainingDrQ+BC · V-D4RL Medium Cheetah Runaverage episode return: 340.33±7.55 -> 379.80±45.83Improved
add transition-prediction pretrainingDrQ+BC · V-D4RL Medium Replay Walker walkaverage episode return: 30.09±0.75 -> 28.68±2.29Hurt
add transition-prediction pretrainingTD3+BC · D4RL Medium HalfCheetah, 10% dataproxy Q-error: 503.6851 -> 287.7740Improved
add transition-prediction pretrainingTD3+BC · D4RL Medium Replay Hopper, 10% dataproxy Q-error: 320.1790 -> 319.9531No clear effect
add transition-prediction pretrainingAWAC · Adroit Human Doornormalized score: 60.82±12.38 -> 29.96±22.43Hurt
freeze the pretrained backbone during RL training (ablation)TD3+BC + Ours · D4RL Random Hoppernormalized score: 11.0±0.1 -> 11.59±10.56No clear effect

Experimental setup

Datasets: D4RL Robomimic V-D4RL ExoRL Adroit

Benchmarks: D4RL Robomimic V-D4RL ExoRL Adroit

Metrics: normalized score success rate episode return proxy Q-error latent feature rank

Baselines: AWAC CQL IQL TD3+BC IRIS BCQ DrQ+BC MOPO MOBILE ACL TD3

Builds on: TD3+BC AWAC CQL IQL DrQ+BC

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy