Research map / Reinforcement learning / Policy gradient and actor-critic
Revisiting Fundamentals of Experience Replay
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase replay capacity from 1M to 10M while fixing replay ratio | DQN · Atari | relative performance: +0.1% | No clear effect |
| Increase replay capacity from 1M to 10M while fixing oldest policy | DQN · Atari | relative performance: -0.4% | No clear effect |
| Increase replay capacity from 1M to 10M while fixing replay ratio | Rainbow · Atari | relative performance: +28.7% | Helped |
| Increase replay capacity from 1M to 10M while fixing oldest policy | Rainbow · Atari | relative performance: +18.3% | Helped |
| Increase replay capacity | Rainbow · Atari 14-game subset | performance | Helped |
| Reduce the age of the oldest policy in replay | Rainbow · Atari 14-game subset | performance | Mixed |
| Add n-step returns (ablation) | DQN · Atari 20-game subset | relative improvement from increasing replay capacity | Helped |
| Remove n-step returns (ablation) | Rainbow · Atari 20-game subset | relative improvement from increasing replay capacity | Hurt |
| Remove prioritized experience replay (ablation) | Rainbow · Atari 20-game subset | relative improvement from increasing replay capacity | Helped |
| Remove Adam (ablation) | Rainbow · Atari 20-game subset | relative improvement from increasing replay capacity | Helped |
| Remove C51 (ablation) | Rainbow · Atari 20-game subset | relative improvement from increasing replay capacity | Helped |
| Use a smaller bootstrap contraction factor in a 1-step update instead of n-step returns (ablation) | DQN · Atari | relative improvement from increasing replay capacity | No clear effect |
Closest papers by meaning
- Remember and Forget for Experience Replay (ICML 2019 · similarity 0.93)
- Variance Reduction via Resampling and Experience Replay (AAAI 2026 · similarity 0.92)
- Attentive Experience Replay (AAAI 2020 · similarity 0.92)
- When Does Non-Uniform Replay Matter in Reinforcement Learning? (NeurIPS 2026 · similarity 0.90)
- A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control (ICML 2025 · similarity 0.88)
- The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.