Research map / Reinforcement learning / Visual and representation RL
Investigating Memory in Model-Free RL with POPGym Arcade
Zekang Wang, Zhe He, Borong Zhang, Edan Toledo, Steven Morad
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compare POMDP return only vs Observability Gap and Memory Bias | GDN vs MinGRU · POPGym Arcade (aggregated) | return, Gap, Bias: POMDP return 0.469 vs 0.471; Gap 0.090 smaller, Bias 0.091 more negative for GDN | Mixed |
| add memory model to a policy on fully observable tasks | memory-endowed policies · POPGym Arcade MDPs | Memory Bias | Hurt |
| increase number of memory layers beyond a point (ablation) | recurrent memory models · BattleShip and MineSweeper | Memory Bias | Hurt |
| increase recurrent state size (ablation) | recurrent memory models · BattleShip and MineSweeper | Observability Gap | Improved |
| train agents with PQN (memory bypass skip connection included) | memory models · POPGym Arcade MDPs | Recall density on early trajectory | Hurt |
| set discount factor gamma to 0 to mimic model-based reward prediction (ablation) | PQN with LRU · CartPole MDP | value smearing | No clear effect |
| replace PQN with PPO or DQN (ablation) | memory models · POPGym Arcade Easy tasks | value smearing, negative Memory Bias | No clear effect |
| inject noise into one past observation | memory-endowed policies · POPGym Arcade | policy robustness | Hurt |
| shuffle first observations of trajectory to remove CNN confounders | LRU and Transformer policies · BattleShip, MineSweeper, CartPole | policy robustness | Hurt |
| longer rollout horizons after noisy frame | LRU · CartPole | contamination persistence: 50 and 100 timesteps | Hurt |
Closest papers by meaning
- Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Partially Observable Reinforcement Learning with Memory Traces (ICML 2025 · similarity 0.87)
- Mastering Memory Tasks with World Models (ICLR 2024 · similarity 0.86)
- Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.86)
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
