AI Paper Atlas

Research map / Reinforcement learning / Offline RL

The Difficulty of Passive Learning in Deep Reinforcement Learning

Georg Ostrovski, Pablo Samuel Castro, Will Dabney

NeurIPS 2021 · Reinforcement learning › Offline RL · approach: Simple practical offline RL · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use active agent's target Q and/or policy for passive bootstrapping targets (ablation)passive agent in Tandem DQN · Tandem DQN, Ataripassive/active performanceNo clear effect
raise the active agent's epsilon-greedy exploration (more coverage of non-argmax actions)Tandem DQN · Tandem DQN, Atarirelative passive performanceImproved
use sticky actions in Atari (environment-side stochasticity)Tandem DQN · Tandem DQN, Atarirelative passive performanceNo clear effect
increase the passive agent's replay buffer size beyond 1MTandem DQN · Tandem DQN, Ataripassive performanceMixed
freeze the active policy and train the passive agent on its data (forked tandem)Forked Tandem DQN · Tandem DQN, Ataripassive performanceHurt
freeze the replay buffer and keep training the passive agentForked Tandem DQN · Tandem DQN, Ataripassive performanceHurt
replace Q-learning with on-policy SARSA evaluation in forked tandem (ablation)Forked Tandem DQN · Tandem DQN, Ataripassive performanceNo clear effect
mix passive agent's own generated data into its replay batches (10%-20%)Tandem DQN · Tandem DQN, Ataritandem effectImproved
mix 50/50 own data with active data into passive replay batchesTandem DQN · Tandem DQN, Ataritandem effectImproved
switch optimizer from RMSProp to Adam (ablation)Tandem DQN · Tandem DQN, Ataritandem effectNo clear effect
increase the number of passive updates per active update (ablation)passive agent in Tandem DQN · Tandem DQN, Ataripassive performanceHurt
replace TD training with regression onto the active network's outputs (distillation) (ablation)passive agent in Tandem DQN · Tandem DQN, Atariperformance gapImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy