Research map / Reinforcement learning / Policy gradient and actor-critic
The Surprising Difficulty of Search in Model-Based Reinforcement Learning
Wei-Di Chang, Mikael Henaff, Brandon Amos, Gregory Dudek, Scott Fujimoto
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MPC to MR.Q | MR.Q · Gym | normalized score: 1.46 -> 0.67 | Hurt |
| add MPC to MR.Q | MR.Q · DMC | average return / 1k: 0.84 -> 0.65 | Hurt |
| add MPC to MR.Q | MR.Q · HumanoidBench with hand | average return / 1k: 0.31 -> 0.38 | Improved |
| use MRS.Q (min over 10 value functions plus MPC) instead of MR.Q + MPC | MR.Q + MPC · HumanoidBench with hand | average return / 1k: 0.38 -> 0.58 | Improved |
| use 10 value functions instead of 2 (ablation) | MRS.Q · Gym | normalized score difference: -0.63 | Improved |
| use 10 value functions instead of 5 (ablation) | MRS.Q · Gym | normalized score difference: -0.37 | Improved |
| use 20 value functions instead of 10 (ablation) | MRS.Q · HumanoidBench no hand | performance difference: 0.03 | No clear effect |
| take min over the full ensemble instead of two sampled members (ablation) | MRS.Q · Gym | normalized score difference: -0.49 | Improved |
| take min over full ensemble during MPC instead of mean (ablation) | MRS.Q · Gym | normalized score difference: -0.33 | Improved |
| remove exploration noise (ablation) | MRS.Q · Gym | normalized score difference: -0.13 | Improved |
| use SEM instead of ELU (ablation) | MRS.Q · Gym | normalized score difference: -0.28 | Improved |
| use SEM instead of ELU (ablation) | MRS.Q · HumanoidBench with hand | performance difference: 0.10 | Hurt |
Closest papers by meaning
- The Value Equivalence Principle for Model-Based Reinforcement Learning (NeurIPS 2020 · similarity 0.90)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.88)
- Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.88)
- Query-Policy Misalignment in Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
