Research map / Reinforcement learning / Offline RL
Efficient Online Reinforcement Learning with Offline Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LayerNorm to critic (RLPD design choice) (ablation) | RLPD on Sparse Adroit (expert-only data) · Expert Adroit Sparse Tasks | normalized score | Helped |
| Add LayerNorm to critic (ablation) | RLPD on Adroit Sparse (full data) · Sparse Adroit (Pen, Door, Relocate) | variance across seeds / mean score | Mixed |
| Add symmetric sampling (50% online / 50% offline batches) (ablation) | RLPD vs buffer initialization · Adroit Human Expert Demonstrations (Pen, Door) | sample efficiency and stability | Helped |
| Initialize replay buffer with large volume of offline data instead of symmetric sampling (ablation) | SAC on D4RL locomotion with medium data · D4RL Locomotion (medium) | asymptotic performance | Hurt |
| Use critic ensembling as regularizer (vs weight decay and dropout) (ablation) | RLPD critic regularization · AntMaze, Adroit, Locomotion subset | normalized score | Helped |
| Use Dropout as critic regularizer (ablation) | RLPD critic regularization · Sparse reward tasks (AntMaze, Adroit) | normalized score | Hurt |
| Use weight decay (0.01) as critic regularizer (ablation) | RLPD critic regularization · AntMaze, Adroit, Locomotion subset | normalized score | Hurt |
| Increase sampling proportion of offline data to 25% (ablation) | RLPD sampling proportion · walker2d-medium-v0 and sparse reward tasks | asymptotic performance / variance | Mixed |
| Use 2-member critic ensemble instead of 10 in pixel tasks (ablation) | RLPD on V-D4RL · V-D4RL Humanoid Walk | normalized return | Hurt |
| Use clipped double Q-learning (CDQ) on AntMaze Large Diverse (ablation) | RLPD per-environment choices · AntMaze Large Diverse | normalized score | Hurt |
| Use CDQ on Humanoid Walk Expert (ablation) | RLPD per-environment choices · V-D4RL Humanoid Walk Expert | normalized return | Hurt |
| Add entropy backups and use smaller networks (2 layer) in RLPD per-environment choices (ablation) | RLPD hardest tasks · Relocate, AntMaze Large Diverse, Humanoid Expert | normalized score | Hurt |
Closest papers by meaning
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.94)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.93)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.93)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.92)
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.92)
- Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.