Research map / Reinforcement learning / Policy gradient and actor-critic
Diminishing Return of Value Expansion Methods
Daniel Palenicek, Michael Lutter, Joao Carvalho, Daniel Dennert, Faran Ahmad, Jan Peters
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase rollout horizon beyond 5 steps | SAC critic/actor expansion (oracle dynamics) · Brax continuous control (aggregated) | IQM normalized return: peak performance at most H=5; no further increase for more than 5 steps | No clear effect |
| use oracle dynamics instead of learned ensemble | SAC-CE · Brax continuous control (aggregated) | final performance: up to ~20% higher final performance | Improved |
| use oracle dynamics instead of learned ensemble | SAC-AE · Brax continuous control (aggregated) | final performance and learning speed: neural network model learns slightly faster | Hurt |
| increase rollout horizon for actor expansion | SAC-AE · Brax continuous control | return: performance degrades significantly for longer horizons | Hurt |
| replace learned model rollouts with model-free Retrace | SAC-CE (learned model) · Brax continuous control (aggregated) | sample efficiency / wall-clock: up to 15x faster in wall-clock time at H=5 | No clear effect |
| use deterministic policy (DDPG) so target variance does not grow with H (ablation) | DDPG-CE / DDPG-AE · Brax continuous control | return: same diminishing returns as SAC | No clear effect |
| average Q-targets over particles (30 for CE, 10 for AE) (ablation) | SAC-CE and SAC-AE · Brax continuous control | return: no qualitative difference in training performance | No clear effect |
| vary discount factor gamma (ablation) | SAC value expansion · InvertedPendulum | return: diminishing returns obvious in all experiments regardless of gamma | No clear effect |
| increase rollout horizon of oracle model | DQN-CE and DQN-Retrace · MinAtar (4 games, aggregated) | return: do not benefit from longer rollout horizons | No clear effect |
| increase rollout horizon on target distribution | SAC H-step target · Brax continuous control | Wasserstein distance to true return: distance decreases with horizon; target variance increases by orders of magnitude | Improved |
| increase horizon on critic gradients | SAC-CE · Brax continuous control | critic gradient std: std between 10^-3 and 10^-1 | No clear effect |
| increase horizon on actor gradients | SAC-AE · HalfCheetah | actor gradient std: explodes for horizons >=3 (oracle) and 20 (learned) | Hurt |
Closest papers by meaning
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Sample-Efficiency in Multi-Batch Reinforcement Learning: The Need for Dimension-Dependent Adaptivity (ICLR 2024 · similarity 0.88)
- Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates (ICLR 2024 · similarity 0.87)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.87)
- Maximum Entropy Model Correction in Reinforcement Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
