Research map / Reinforcement learning / Policy gradient and actor-critic
When to use parametric models in reinforcement learning?
Hado P van Hasselt, Matteo Hessel, John Aslanides
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use data-efficient Rainbow DQN instead of SimPLe | Atari 26 games · Atari 26 games | median human-normalized episode return: roughly $25\%$ higher performance by 100,000 interactions; 17 out of 26 games | Improved |
| Increase multi-step return length from 3 to 20 (ablation) | Rainbow DQN · Atari 26 games | median human-normalized episode return | Improved |
| Use a 5-step rather than 20-step bootstrapping length (ablation) | Rainbow DQN · Atari 26 games | median human-normalized episode return | Hurt |
| Use a 10-step rather than 20-step bootstrapping length (ablation) | Rainbow DQN · Atari 26 games | median human-normalized episode return | No clear effect |
| Perform updates every step rather than every 4 steps (ablation) | Rainbow DQN · Atari 26 games | median human-normalized episode return | Improved |
| Increase replay batch size instead of updating every step, matching the 4× increase in replayed transitions (ablation) | Rainbow DQN · Atari 26 games | median human-normalized episode return | Hurt |
| Plan backward rather than forward for credit assignment | Tabular Dyna in four-rooms · Four-rooms, deterministic and stochastic variants | episode length | Improved |
| Use replay instead of forward model planning for credit assignment | Tabular Q-learning in four-rooms · Four-rooms, deterministic variant | episode length: after 2000 episodes | Mixed |
Closest papers by meaning
- The Benefits of Model-Based Generalization in Reinforcement Learning (ICML 2023 · similarity 0.89)
- Prioritized Model Experience Replay (ICML 2026 · similarity 0.89)
- Revisiting Fundamentals of Experience Replay (ICML 2020 · similarity 0.88)
- When Does Non-Uniform Replay Matter in Reinforcement Learning? (NeurIPS 2026 · similarity 0.87)
- Prioritized Generative Replay (ICLR 2025 · similarity 0.86)
- Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
