Research map / Reinforcement learning / Policy gradient and actor-critic
Reconciling λ-Returns with Experience Replay
Brett Daley, Christopher Amato
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Peng's Q(λ) returns instead of 3-step returns | DQN · Six Atari 2600 games | game score | Mixed |
| Use Peng's Q(λ) with λ=0.25 or 0.5 | DQN · Breakout and Space Invaders | game score | Improved |
| Use Watkin's Q(λ) instead of Peng's Q(λ) (ablation) | DQN(λ) · Six Atari 2600 games | game score | Hurt |
| Use median-based dynamic λ selection rather than the best hand-picked λ (ablation) | DQN(λ) · Six Atari 2600 games | game score | Mixed |
| Use error-bounded dynamic λ selection instead of median-based selection (ablation) | DQN(λ) · Breakout, Space Invaders, Beam Rider, Q*Bert, and Seaquest | game score | Mixed |
| Use the cache-based DQN(λ) method with the best tested λ instead of target-network 3-step DQN | DQN · Five of six Atari 2600 games | training wall-clock time | Improved |
| Use median-based dynamic λ selection with one-frame inputs (ablation) | DQN(λ) · Six Atari 2600 games under partial observability | relative performance versus 3-step DQN | No clear effect |
| Increase cache size from the target-network comparison to larger caches (ablation) | 3-step DQN with cache · Seaquest | game score | Improved |
| Use cache-based replay rather than target-network replay (ablation) | 3-step DQN · Seaquest | game score | Hurt |
| Use cache-based replay rather than target-network replay (ablation) | 3-step DQN · Space Invaders | game score | Improved |
Closest papers by meaning
- Revisiting Fundamentals of Experience Replay (ICML 2020 · similarity 0.91)
- When Does Non-Uniform Replay Matter in Reinforcement Learning? (NeurIPS 2026 · similarity 0.87)
- Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound (NeurIPS 2025 · similarity 0.87)
- The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning (ICML 2025 · similarity 0.87)
- Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization (ICML 2024 · similarity 0.86)
- Simplifying Deep Temporal Difference Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
