Research map / Reinforcement learning / Offline RL
The Difficulty of Passive Learning in Deep Reinforcement Learning
Georg Ostrovski, Pablo Samuel Castro, Will Dabney
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use active agent's target Q and/or policy for passive bootstrapping targets (ablation) | passive agent in Tandem DQN · Tandem DQN, Atari | passive/active performance | No clear effect |
| raise the active agent's epsilon-greedy exploration (more coverage of non-argmax actions) | Tandem DQN · Tandem DQN, Atari | relative passive performance | Improved |
| use sticky actions in Atari (environment-side stochasticity) | Tandem DQN · Tandem DQN, Atari | relative passive performance | No clear effect |
| increase the passive agent's replay buffer size beyond 1M | Tandem DQN · Tandem DQN, Atari | passive performance | Mixed |
| freeze the active policy and train the passive agent on its data (forked tandem) | Forked Tandem DQN · Tandem DQN, Atari | passive performance | Hurt |
| freeze the replay buffer and keep training the passive agent | Forked Tandem DQN · Tandem DQN, Atari | passive performance | Hurt |
| replace Q-learning with on-policy SARSA evaluation in forked tandem (ablation) | Forked Tandem DQN · Tandem DQN, Atari | passive performance | No clear effect |
| mix passive agent's own generated data into its replay batches (10%-20%) | Tandem DQN · Tandem DQN, Atari | tandem effect | Improved |
| mix 50/50 own data with active data into passive replay batches | Tandem DQN · Tandem DQN, Atari | tandem effect | Improved |
| switch optimizer from RMSProp to Adam (ablation) | Tandem DQN · Tandem DQN, Atari | tandem effect | No clear effect |
| increase the number of passive updates per active update (ablation) | passive agent in Tandem DQN · Tandem DQN, Atari | passive performance | Hurt |
| replace TD training with regression onto the active network's outputs (distillation) (ablation) | passive agent in Tandem DQN · Tandem DQN, Atari | performance gap | Improved |
Closest papers by meaning
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
- Neural Network Approximation for Pessimistic Offline Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Hybrid Reinforcement Learning from Offline Observation Alone (ICML 2024 · similarity 0.89)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.89)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
