AI Paper Atlas

Research map / Reinforcement learning / Visual and representation RL

Investigating Memory in Model-Free RL with POPGym Arcade

Zekang Wang, Zhe He, Borong Zhang, Edan Toledo, Steven Morad

ICML 2026 · Reinforcement learning › Visual and representation RL · approach: Scalable RL systems and environments

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
compare POMDP return only vs Observability Gap and Memory BiasGDN vs MinGRU · POPGym Arcade (aggregated)return, Gap, Bias: POMDP return 0.469 vs 0.471; Gap 0.090 smaller, Bias 0.091 more negative for GDNMixed
add memory model to a policy on fully observable tasksmemory-endowed policies · POPGym Arcade MDPsMemory BiasHurt
increase number of memory layers beyond a point (ablation)recurrent memory models · BattleShip and MineSweeperMemory BiasHurt
increase recurrent state size (ablation)recurrent memory models · BattleShip and MineSweeperObservability GapImproved
train agents with PQN (memory bypass skip connection included)memory models · POPGym Arcade MDPsRecall density on early trajectoryHurt
set discount factor gamma to 0 to mimic model-based reward prediction (ablation)PQN with LRU · CartPole MDPvalue smearingNo clear effect
replace PQN with PPO or DQN (ablation)memory models · POPGym Arcade Easy tasksvalue smearing, negative Memory BiasNo clear effect
inject noise into one past observationmemory-endowed policies · POPGym Arcadepolicy robustnessHurt
shuffle first observations of trajectory to remove CNN confoundersLRU and Transformer policies · BattleShip, MineSweeper, CartPolepolicy robustnessHurt
longer rollout horizons after noisy frameLRU · CartPolecontamination persistence: 50 and 100 timestepsHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy