Research map / Reinforcement learning / Visual and representation RL
COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COPlanner, a plug-and-play planning framework that uses uncertainty-aware policy-guided MPC to penalize model uncertainty during model rollouts (conservative) and reward it during real-environment exploration (optimistic), improving MBPO and DreamerV3.
Problem
Dyna-style model-based RL suffers from dynamics model errors that mislead policy learning from model rollouts, while exploration to reduce model error is sample-inefficient or ignores reward.
Approach
Use policy-guided MPC over H_p steps with random shooting to accumulate reward plus/minus multi-step ensemble-disagreement uncertainty; the sign of the uncertainty term is negative for model rollouts (conservative) and positive for real exploration (optimistic), using one policy without intrinsic rewards.
Key results
- COPlanner-MBPO improves sample efficiency of MBPO by about 40% on proprioceptive DMC and almost doubles it on GYM tasks; performance improves 16.9% on DMC and 32.8% on GYM at convergence.
- COPlanner-DreamerV3 improves performance of DreamerV3 by 9.6% on 8 visual DMC tasks with more than twice the sample efficiency.
- Ablation shows combining optimistic exploration and conservative rollouts gives the best results; using either alone yields smaller gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add COPlanner to MBPO (optimistic environment exploration plus conservative model rollouts) | MBPO · Proprioceptive DMC | sample efficiency: 40% | Helped |
| Add COPlanner to MBPO (optimistic environment exploration plus conservative model rollouts) | MBPO · Proprioceptive DMC | performance: 16.9% | Helped |
| Add COPlanner to MBPO | MBPO · GYM | sample efficiency: almost doubled | Helped |
| Add COPlanner to MBPO | MBPO · GYM | performance: 32.8% | Helped |
| Add COPlanner to DreamerV3 | DreamerV3 · Visual DMC | sample efficiency: more than twice | Helped |
| Add COPlanner to DreamerV3 | DreamerV3 · Visual DMC | performance: 9.6% | Helped |
| Use optimistic exploration only, without conservative rollouts (ablation) | COPlanner · Walker-stand, Walker-run, Hopper-hop, Cartpole-swingup-sparse | sample efficiency and performance | Helped |
| Use conservative rollouts only, without optimistic exploration (ablation) | COPlanner · Walker-stand, Walker-run, Hopper-hop, Cartpole-swingup-sparse | sample efficiency and performance | Mixed |
| Combine optimistic exploration with conservative rollouts (ablation) | COPlanner · Walker-stand, Walker-run, Hopper-hop, Cartpole-swingup-sparse | performance | Helped |
| Replace model disagreement uncertainty with RE3 (ablation) | COPlanner-DreamerV3 · Hopper-hop | performance | No clear effect |
| Replace model disagreement uncertainty with MADE (ablation) | COPlanner-DreamerV3 · Hopper-hop | performance | No clear effect |
| Set planning horizon to 1 instead of the default multi-step horizon (ablation) | COPlanner-MBPO and COPlanner-DreamerV3 · Walker-run, Cheetah-run, Hopper-hop | performance and sample efficiency | Hurt |
Surprising findings
- Conservative rollouts alone can over-conserve in visual tasks, causing failure on a sparse-reward seed (Cartpole-swingup-sparse).
- Using a one-step planning horizon (H_p=1) gives only limited improvement, showing multi-step uncertainty matters.
Limitations
- MPC planning adds computational time relative to base methods.
- Conservative rollouts alone can be overly conservative in sparse-reward visual tasks.
- Hyperparameters (alpha_o, alpha_c, K, H_p) are task-sensitive and tuned per domain.
Experimental setup
Datasets: DeepMind Control Suite MuJoCo-GYM
Benchmarks: DeepMind Control Suite (proprioceptive) MuJoCo GYM (Hopper, Walker2d, Ant, Humanoid) DeepMind Control Suite (visual)
Metrics: episode return sample efficiency state entropy
Baselines: MBPO P2P-MPC MEEE M2AC D4PG DreamerV3 LEXA-reward-DreamerV3 DrQv2 TDMPC PlaNet PDML MoPAC
Builds on: MBPO DreamerV3 Plan2Explore random shooting MPC MEEE
Closest papers by meaning
- COMBO: Conservative Offline Model-Based Policy Optimization (NeurIPS 2021 · similarity 0.90)
- Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization (ICLR 2023 · similarity 0.89)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Optimistic Model Rollouts for Pessimistic Offline Policy Optimization (AAAI 2024 · similarity 0.88)
- SOMBRL: Scalable and Optimistic Model-Based RL (NeurIPS 2025 · similarity 0.87)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.