Research map / Reinforcement learning / Policy gradient and actor-critic
Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STEVE's uncertainty-aware weighting of model-rollout horizons | DDPG on continuous-control benchmarks · Humanoid-v1, RoboschoolHumanoid-v1, RoboschoolHumanoidFlagrun-v1, and BipedalWalkerHardcore-v2 | learning within 5M frames: within 5M frames | Helped |
| Use STEVE's uncertainty-aware weighting of model-rollout horizons | Tabular value estimation with a noisy dynamics model · Tabular toy environment | speedup over TD learning: 2× speedup over TD learning | Helped |
| Use model-based value expansion with a perfect dynamics model | Tabular TD learning · Tabular toy environment | samples to solve: 5× fewer samples | Helped |
| Use a noisy model with a fixed MVE rollout horizon | Tabular MVE · Tabular toy environment | value-estimation error | Hurt |
| Replace STEVE's uncertainty-aware weighting with uniform weighting (ablation) | STEVE architecture (Mean-MVE) · Humanoid-v1 | sample efficiency | Hurt |
| Replace STEVE's uncertainty-aware weighting with TD(λ) weighting at λ=0.25 (ablation) | STEVE architecture (TDL25) · Humanoid-v1 | sample efficiency | Hurt |
| Replace STEVE's uncertainty-aware weighting with TD(λ) weighting at λ=0.75 (ablation) | STEVE architecture (TDL75) · Humanoid-v1 | sample efficiency | Hurt |
| Include covariance terms when computing target weights (ablation) | STEVE (COV-STEVE) · Humanoid-v1 | performance | Hurt |
| Increase the rollout horizon (ablation) | STEVE · Humanoid-v1 | sample efficiency | Helped |
| Use Bayesian neural networks or dropout instead of ensembles for uncertainty estimates (ablation) | STEVE's model and value-function uncertainty estimation · Continuous-control experiments | base-model accuracy | Hurt |
Closest papers by meaning
- Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning (ICLR 2023 · similarity 0.91)
- Model-Free Active Exploration in Reinforcement Learning (NeurIPS 2023 · similarity 0.89)
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (NeurIPS 2018 · similarity 0.89)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Improving Sample Efficiency of Model-Free Algorithms for Zero-Sum Markov Games (ICML 2024 · similarity 0.87)
- Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.