Research map / Reinforcement learning / Policy gradient and actor-critic
Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase rollout horizon from 1 to 5 steps (diminishing returns beyond 5) | SAC critic and actor expansion with oracle and learned models · InvertedPendulum, Cartpole, Hopper, Walker2d, HalfCheetah | sample efficiency / episodic return | Helped |
| increase rollout horizon beyond 5 steps | SAC critic and actor expansion with oracle dynamics · five continuous control benchmarks | sample efficiency / episodic return | No clear effect |
| use oracle dynamics model instead of learned ensemble model (ablation) | SAC critic expansion and actor expansion · five continuous control benchmarks | sample efficiency / episodic return | No clear effect |
| use oracle dynamics model instead of learned ensemble model (ablation) | SAC actor expansion at long horizons · HalfCheetah and Hopper | actor gradient stability and return | No clear effect |
| average Q-targets over multiple particles (30 for CE, 10 for AE) to reduce target variance (ablation) | SAC critic and actor expansion · five continuous control benchmarks | training performance | No clear effect |
| use deterministic policy (DDPG) instead of stochastic SAC (ablation) | value expansion with oracle dynamics · five continuous control benchmarks | diminishing returns of horizon | No clear effect |
| use model-free Retrace instead of model-based value expansion | SAC critic expansion · five continuous control benchmarks | sample efficiency | No clear effect |
| use oracle model vs Retrace (model-free) at H=5 for wall-clock cost | model-based critic and actor expansion · wall-clock training time | training time: up to 15x slower | Hurt |
| use actor expansion instead of critic expansion (backprop through rollout) | SAC value expansion · HalfCheetah, Hopper, Walker2d | return and gradient stability | Hurt |
| increase discount-driven horizon via discount factor change (ablation) | SAC value expansion on InvertedPendulum · InvertedPendulum | diminishing returns pattern | No clear effect |
Closest papers by meaning
- Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion (NeurIPS 2018 · similarity 0.91)
- Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization (ICLR 2023 · similarity 0.89)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Sample-Efficiency in Multi-Batch Reinforcement Learning: The Need for Dimension-Dependent Adaptivity (ICLR 2024 · similarity 0.87)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.87)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.